Poor Man’s Data Munging in the Linux Shell

This is just a brief cheat sheet style of post. While there are many sophisticated tools for processing data out there, sometimes, it may be more convenient or quicker to just use the tools the Linux shell provides (Please note that I am working in bash and don’t know how much of this applies to other shells.). In the following, I primarily post a brief overview of some commands I found useful for the quick and dirty munging of data.


# Filtering/Selecting Data
########################

# Filter by row/line:
grep EXPRESSION FILE

# Filter by column:
awk FILE '{print $3, $5}'

# Filter by column, with custom separator, e.g., ',':
awk -F, FILE '{print $3, $5}'

# Filter by column, with custom output separator, e.g., ',':
awk FILE 'BEGIN {OFS=","} {print $3, $5}'

# Transformation
##############

# Add line numbers:
nl -w1 FILE
# Note this is 1 (one) not a lowercase 'L'.

# Add line numbers with custom separator, e.g., ',':
nl -s, -w1 FILE

# Sort lexicographically:
sort FILE

# Sort numerically:
sort -n FILE

# "Transpose" a file, more precisely output content as single row:
tr -d '\n' < FILE

# Get first n (e.g. 10) rows:
head -n 10 FILE

# Get last n (e.g. 10) rows:
tail -n 10 FILE

# Omit the first n (e.g. 10) rows (Get all but the first n rows.):
tail -n +10 FILE

# Omit the last n (e.g. 10) rows (Get all but the last n rows.):
head -n -10 FILE

# head and tail also work character-wise (Beware of the trailing '\n'.):
echo "ABCDE" | tail -c 2
#> E
echo "ABCDE" | tail -c +2
#> BCDE
echo "ABCDE" | head -c 2
#> AB
echo "ABCDE" | head -c -2
#> ABCD

# Add leading zeros:
printf "%03d" 7

# Remove leading zeros:
printf "%d" 007

# Output
######

# Repeat string (e.g. "foo\n") n times (e.g., 10):
printf 'foo\n%.0s' $(seq 1 1 10)

# Concatenation
#############

# Concatenate two files (add FILE2 to the end of FILE1):
cat FILE1 FILE2

# Concatenate two files column-wise:
paste FILE1 FILE2

# Concatenate two files column-wise, with custom separator, e.g., ',':
paste -d, FILE1 FILE2

# Simple Calculations
#################

# Integer operations can be done directly in the shell:
echo $((1+2))

# For floating point operations the "bc" tool can be used:
echo "1.23 * 2" | bc

# Simple Statistics
#################

# Calculate min, mean, median, max, sd, for data in column 3:
awk FILE '{print $3}' | Rscript -e 'd<-scan("stdin", quiet=TRUE); cat(min(d), mean(d), median(d), max(d), sd(d), sep=" ")'

“Rscript” as seen in the last example can generally be used for invoking R.

Another way for invoking R that I found useful is in shell scripts via here files. The following snippet shows a simple script that iterates over all *.raw.data files in a directory, creates a box plot, and stores the boxplot in a *.png file:


#!/bin/bash

for f in DIRECTORY/*.raw.data

do

R --slave --vanilla --no-save --quiet <<R_END
data <- read.table("$f")
png("${f}.png")
boxplot(data)
dev.off()
R_END

done

Of course, this example assumes that the data in the raw data files is already in a format that can be read by R. Something like this can be handy, e.g., if you want to get a quick overview of a bigger number of data files.

Posted in Snippets | Tagged | Leave a comment

Panoramic Photos, “Beautiful British Columbia”, Canada

In this post, I upload some panoramic photos I took in May during a trip to “Beautiful British Columbia”, Canada, which, I think, truly deserves this byname. Even though I just visited a very small part of British Columbia, I really enjoyed the beautiful nature. Unfortunately, I didn’t get around to uploading these photos earlier.

The first photo shows Brohm Lake, which is located some driving minutes north of Squamish.

brohm_lake

Squamish itself is a small town located about an hour drive north of Vancouver. There are many beautiful places around Squamish. The following picture shows the Brandy Wine Falls located about 30 minutes north of Squamish.

brandywine_falls

The next two photos are no panoramas. I add them anyhow as I think these are really beautiful. The photos show the Shannon Falls which are located a few minutes south of Squamish.

The next photo shows a view from Porteau Cove, which is located about 15 minutes south of Squamish.

porteau_cove

In the following panorama, a view from the Tantalus view point at the Sea to the Sky Highway is shown. This view point is located about 15 minutes north of Squamish. Generally, there are many nice viewing points along the Sea to the Sky Highway.

tantalus_view_point

The next panorama does not show nature but the inside of the main building of the Brittania Mining Museum, located about 15 minutes south of Squamish. I added this photo as I think it shows an impressive sight as well.

brittania_mining_museum

The next two pictures were taken on Vancouver Island. The first shows the view from the Holland Point in Victoria. The second shows flowers in Butchard Gardens.

victoria_holland_point

butchart_gardens_1

The last three pictures were taken in Vancouver. The first shows a view from the Ambleside Park. The last two were taken in Lynn Canyon.

vancouver_sea_at_ambleside_park

lynn_canyon_1

lynn_canyon_2

 

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Posted in Misc. | Tagged , | Leave a comment

#Pebble App for #SailfishOS / #Jolla Updated: SkippingStones 0.17

Recently, I have been even more busy than usually. As one consequence, I couldn’t really keep track of announcing improvements in SkippingStones. In this post, I try to summarize the most important changes very briefly and add a link to the latest SkippingStones version. Great thanks also to Uladzislau Vasilyeu (aka vasvlad) for his improvements and pull requests :).

Below, is just an enumeration of the improvements in no particular order:

  • Ability to Manage Watchfaces and Watchapps (Please use this with caution as it happened to me that the watch was set into recovery mode and I had to recover it via the Android app.)
  • Support for some SmartStatus Watchapps (see below)
  • Support for Media Player Controls including Volume via SmartStatus Watchapps
  • Display of Current Weather Icon in SmartStatus with MeeCast (Requires a patch.)
  • Display the Phone Battery Level in SmartStatus
  • Possibility Hangup for PhoneControl
  • Support for “CalendarReminder”
  • Show Name from adressbook for PhoneNumber in incoming call message
  • Fixed problem with non-Latin letters in notification about incoming call

I hope I didn’t forget any important change. Also, I don’t have the time, right now, to add more detailed explanations, e.g., on applying the MeeCast patch etc. For more details etc. see the thread linked below.

Please note that SkippingStones comes without any warranty etc. The new version can be downloaded from: https://github.com/ruedigergad/SkippingStones/blob/master/dist/harbour-skippingstones-0.17-1.armv7hl.rpm?raw=true

So far, I tested the SmartStatus support with these watchapps: http://www.mypebblefaces.com/apps/21105/10414/ http://www.mypebblefaces.com/apps/4805/6490/6508/

If there are no critical errors in this version, I plan to upload it to openrepos as well. However, for this, I need your feedback. ;)

For more information and discussion, please see the following thread: http://talk.maemo.org/showthread.php?t=92695

Posted in My Applications, SkippingStones | Tagged , , , , , | Leave a comment

#Pebble App for #SailfishOS / #Jolla Updated: SkippingStones 0.10

In the last days, I put a little bit more work into my SkippingStones app that enables the usage of Pebble smart watches with Jolla smartphones. The result is SkippingStones version 0.10 which features the following improvements.

  • Improved Handling of Notifications (e.g., E-Mail, XMPP, Whatsapp (Mitakuuluu), etc.)
  • Auto-connect Feature
  • Simple App Cover
  • Fixed Unicode Support
  • New Icon

Furthermore, I uploaded SkippingStones to OpenRepos. Please also see this thread at talk.maemo.org in which the most discussion about SkippingStones happens right now.

Posted in Announcements, My Applications, SkippingStones | Tagged , , , , | Leave a comment

New Panoramic Photos

Below are two new panoramic photos. The first one was taken some time ago in Palmengarten Frankfurt.

20140301_006 - 20140301_018

The second shows clouds in the view from Bad Vilbel Heilsberg in the directions Frankfurt and Taunus.

20140408_103 - 20140408_125_c_cropped_3

Posted in Misc. | Tagged , | Leave a comment

Control #crazyflie with #zeemote

Recently, I got a Crazyflie. As I also own two Zeemote JS1 remote controls, using these for controlling the Crazyflie suggested itself to me.

crazyflie_zeemotes

In a previous post, I introduced QZeeControl 2 which enables using Zeemote remote controls, e.g., on Linux Desktop systems. In fact, the main motivation for writing QZeeControl 2 was to control the Crazyflie via the Zeemotes. In this post, I briefly demonstrate how controlling the Crazyflie using the Zeemotes works.

In a nutshell, QZeeControl 2 creates a joystick input device via uinput that can then be used in the Crazyflie PC client application. In the video below, the basic interaction with the Crazyflie application is shown.

So far, I had been flying only a few times but it seems that controlling the Crazyflie this way really works well. Right now, I need to practice more. Hopefully, I can post a video of the actual Crazyflie being flown with the Zeemotes in future.

Also note that I use two Zeemotes because I want to control thrust, yaw, pitch, and roll via the joysticks and thus need four via joystick controllable axes. In other configurations, e.g., with automatic altitude control it could be sufficient to just use a single Zeemote.

Posted in Announcements, Misc., My Applications, QZeeControl | Tagged , , , | Leave a comment

QZeeControl 2: Zeemote JS1 Bluetooth Protocol Implementation in Qt5

The Zeemote JS1 is a small bluetooth joystick remote control. In the past, I already created some apps that allow the usage of the Zeemote JS1, like QZeeControl (for Harmattan/MeeGo on the N9 and N950) and Zeecontrol (for Fremantle on the N900). In this post, I write about QZeeControl 2 which is a Qt5 port/re-write of QZeeControl.

Below are a screenshot and a video of QZeeControl 2 in action:

qzeecontrol2_screenshot

The dark gray area with the red square is used for testing the movement in general. The blue letter “A” to “D” indicate the four buttons on the remote control. The four blue text labels “Up” to “Right” indicate the triggering of the “digital joystick emulation” in QZeeControl 2. This emulation, in a nutshell, triggers a key press of a configurable key-code when certain threshold values are exceeded respectively underrun for the corresponding analog joystick axes. Additionally, as can be seen in the video, it is possible to control the mouse with the joystick remote control.

Generally, QZeeControl 2 supports an arbitrary number of remote controls and has configurable key-codes. However, as the current state is a very early prototype, there is no key-code configuration UI yet. At the time of writing, some configuration options can only be set via configuration text files.

The QZeeControl 2 source code can be found at github. I will also upload a tarball with a readily built version. The build will be for Fedora 20.

One of the problems that I encountered while writing QZeeControl 2 in Qt5 was that Bluetooth on Linux in Qt5 is currently only officially supported in combination with Bluez 4. See also another discussions about this topic. Generally, the Bluez API seems to be a source of some confusion.

Fortunately, it seems that the RFCOMM API which is used in QZeeControl for receiving data from the remote control works as-is with Qt5 and Bluez 4 as well as Bluez 5. The only part that had to be taken care of for getting QZeeControl 2 working with Qt5 and Bluez 5 was to re-implement the scanning/discovery part of the app using the new Bluez 5 DBUS API.

In the previous implementations, Xtst was used for interacting with the UI, like triggering key-presses etc. Perspectively, the plan is to port QZeeControl 2 to SailfishOS/Nemomobile as well. As these OS use Wayland, using Xtst is not an option anymore. Thus, in QZeeControl 2, uinput is used instead. Via uinput, QZeeControl, in a nutshell, behaves like a real joystick input device in the system and thus can be used accordingly.

Posted in Announcements, My Applications, QZeeControl | Tagged , , , , , | 1 Comment