One Year with the Friendly Interactive Shell (fish) aka fish-shell

A little bit more than one year ago, I learned about the friendly interactive shell (fish) aka fish-shell and decided to give it a try. Before, I primarily used the Bourne Again Shell (bash) as my main shell. In this post, I write about my experience with fish and some nice tweaks I discovered.

To cut a long story short, I think that fish is really awesome and it helped me to increase my productivity a lot. I am not sure if I will remember to address all the cool things I like about it in this post but I’ll just start and see where I end up. In the following, I will just start with the points that come to mind first and I will outline the features that I consider cool about them.

Automatic Suggestions based on the Command History and Search in the History

When typing something, fish directly starts to provide suggestions of potential completions based on matching the current input to the command history. If a suggestion is useful, just press the right-arrow key and enter to execute it. Otherwise, if you encounter something that looks similar to something that you used before, you can search within the history for matching entries via the up- and down-arrow keys. This matching in the history, like most (or even all) matches in fish, not only matches the beginning of the entries in the history but matches arbitrary sub-strings.

Tab-completion

Fish features pretty clever completions for many “standard” commands. This starts by providing hints and explanations when completing a partially entered commend and goes further to meaningfully completing the corresponding command line arguments.

Another nice completion feature is that the completion is not limited to matching from the start of a string, like bash, but instead it is capable to match sub-strings. I’ll use an example for illustrating this: Say the directory “foo/” contains the files “bar.txt” “bar.sh” “bar.pdf” then one can type:
“ls foo/pdf<TAB>”, which will complete to “foo/bar.pdf”.
This is not restricted to the string at the end of the file but generally matches sub-strings.
So for “bar_abc_123.txt” “bar_def_123.txt” “bar_ghi_123.txt”
“ls foo/def<TAB>” will complete to “foo/bar_def_123.txt”.

Furthermore, you can also use completion with wildcards as well: For a directory with files “123abc.txt” “456abc.xml” “789def.pdf” “123def.pdf”.
Entering: “*a<TAB>” will complete to “*abc.” and will offer the options to complete with “txt” or “xml”.
Entering: “*d<TAB>” will complete to “*def.pdf”.

History Token Search

Another cool feature is that fish lets you go through the previously entered history token-wise. I typically use this when I need to deal with long paths or complex command line arguments. I bound the functionality history-token-search-backward and history-token-search-forward to ALT-up and ALT-down respectively as I use it quite often.

Word-wise Movement Deletion

Yet another cool feature is the ability to perform word-wise movement and deletions when entering commands. This is particularly useful when dealing with long and complex command lines. I put backward-word, forward-word, and backward-kill-word on ALT-left, ALT-right, and ALT-backspace.

History of Visited Directories

Another neat feature is that fish maintains a history of directories that you have been to and allows you to cycle through these directories. This is done via the backward-word and forward-word functionality. As long as nothing was typed yet, using backward-word and forward-word cycles through the list of visited directories.

Git Status in the Prompt

In the Arch Linux Wiki there is also a simple guide on how to enable a nicely looking prompt that shows you the current status of a git repository whenever the current path happens to be inside of a git repository: https://wiki.archlinux.org/index.php/Fish#Prompt

Colors (“Eye Candy”)

Well, you may consider this “just” “eye candy” but from my experience having nicely colored output also helped me when working with the shell. Like many of the things I discussed above, this is most likely something that you have to try on your own for some time in order to get a real idea of its benefits

Caveats

For certain tasks, like writing short snippets of scripts, such as for loops etc. in the shell, be warned that fish does not follow the syntax of shells like bash or sh. It also uses a different way for handling and setting environment variables. This may look like a problem at first, however, for me all the great advantages of fish outweigh the slight overhead of getting used to this minor peculiarity by far.

Summay

I think fish is a truly great shell and that it helped me to become much much more productive. It took me some time to integrate the new capabilities into my workflow but once I got more and more used to the new features, I am really happy about having switched to fish.

In this post, I briefly describe some of the features that I think of as most valuable. All the things I posted are based on my current workflow and experience.

Please let me know, if you happen to know other cool features that I did not mention yet. I am always looking for further improving my workflow.

My Key-bindings

Last but not least, here are my, admittedly pretty simple, fish key-bindings:

~ cat .config/fish/functions/fish_user_key_bindings.fish 
bind \e, 'history-token-search-backward'
bind \e. 'history-token-search-forward'
bind \eup 'history-token-search-backward'
bind \edown 'history-token-search-forward'
bind \ebackspace 'backward-kill-word'
bind \eleft 'backward-word'
bind \eright 'forward-word'

 

Posted in Snippets | Tagged , , | Leave a comment

Dynamic Distributed Remote Packet Capturing with DRePCap

In scope of my PhD as well as my work as researcher, among other things, I also researched the concept of dynamic distribute remote packet capturing. Some results of this work were published in the paper: “Monitoring Traffic in Computer Networks with Dynamic Distributed Remote Packet Capturing,” Ruediger Gad, Martin Kappes, Inmaculada Medina-Bulo, 2015 IEEE International Conference on Communications (ICC), London, UK.

For the evaluation of this approach, I developed the Distributed Remote Packet Capturing (DRePCap) prototype, which was released as Open Source Software. The empirical results in the paper mentioned above were obtained with this prototype.

Other approaches for remotely capturing packets are, e.g., rpcap or packet capturing via SSH. However, these approaches are typically comparably simple in the way that they only explicitly consider single sensor operation and point-to-point connections between remote packet capturing sensor and data receiver. The concept of dynamic distributed remote packet capturing aims on providing more advanced functionality like:

  • providing an overarching infrastructure for network wide remote packet capturing that can be used by an arbitrary number of receivers and with an arbitrary number of sensors,
  • dynamic configuration of packet capturing sensors,
  • dynamic routing of the captured packet data to potentially multiple destinations,
  • or easing packet capturing at multiple remote sensors at the same time.

Furthermore, for improving the performance and usability, the concept is also capable to:

  • employ cooperative sensors for improving the packet capturing performance,
  • self-adaptive sampling for avoiding overload situations at individual sensors,
  • and self-adaptive sensor cooperation in order to ease the usability of operating multiple sensors.

DRePCap consists of four general components:

In the following figure, an overview of the DRePCap architecture is shown:

drepcap_architecture

In this post, I just briefly present the high-level overview of the dynamic distributed remote packet capturing concept and of DRePCap. For more details, please refer to the paper mentioned above, for now. As time permits, I will try to add some more posts about DRePCap as well as the associated sub-projects.

Posted in DRePCap, Libs. | Tagged , , , , | Leave a comment

clj-net-pcap – Improvements and Preparations for next Release

It’s been a while since I posted the last update about clj-net-pcap. Before the work on my PhD thesis started to occupy most of my time, I actually put some more effort into further optimizing and improving clj-net-pcap. However, due to my PhD thesis, I didn’t write about this progress earlier.

In this post, I briefly summarize the work and progress that had been done. In the following, an overview of the new improvements and optimizations is given:

  • Optimized raw data acquisition performance (by a factor of about 5.9)
  • Alternative method for extracting information from raw packet data (performance improvement by a factor of about 2.5)
  • Domain Specific Language (DSL) for dynamically configuring the data extraction
  • Prototype of a self-adaptive mechanism for adjusting the DSL-based data extraction depending on performance constraints
  • Functionality to write extracted data to files and named-pipes (fifos)
  • Multiple output formats (CSV, JSON, ARFF)

The first four of these improvements were also published in the paper: “Improving Network Traffic Acquisition and Processing with the Java Virtual Machine,” Ruediger Gad, Martin Kappes, and Inmaculada Medina-Bulo, 20th IEEE Symposium on Computers and Communications (ISCC), in press. The paper also contains an evaluation of these improvements. The exact versions of clj-net-pcap that were used for the work presented in this paper are marked with corresponding tags.

The performance optimizations of the raw data acquisition performance were done by improving the forwarding of raw packet data from the native libpcap-based parts to the parts inside the JVM. These optimizations include patches in the corresponding JNI-related C and Java code of jNetPcap that is used for interconnecting the native and JVM-based parts. These patches are released under the same license as jNetPcap and are available in the clj-net-pcap repository.

Before the new release will be published, there is still some housekeeping that has to be done. However, the new functionalities can already be used now.

Posted in clj-net-pcap, Libs. | Tagged , , , , | Leave a comment

List of Publications Added

I just added a list of my publications to my website so that anyone who is interested can see the line of research I was and am currently working in. I am also considering to write a short wrap-up about my research and update this website accordingly. However, right now, writing my PhD thesis occupies most of my time. ;)

Posted in Uncategorized | Tagged , , | Leave a comment

Poor Man’s Data Munging in the Linux Shell

This is just a brief cheat sheet style of post. While there are many sophisticated tools for processing data out there, sometimes, it may be more convenient or quicker to just use the tools the Linux shell provides (Please note that I am working in bash and don’t know how much of this applies to other shells.). In the following, I primarily post a brief overview of some commands I found useful for the quick and dirty munging of data.


# Filtering/Selecting Data
########################

# Filter by row/line:
grep EXPRESSION FILE

# Filter by column:
awk FILE '{print $3, $5}'

# Filter by column, with custom separator, e.g., ',':
awk -F, FILE '{print $3, $5}'

# Filter by column, with custom output separator, e.g., ',':
awk FILE 'BEGIN {OFS=","} {print $3, $5}'

# Transformation
##############

# Add line numbers:
nl -w1 FILE
# Note this is 1 (one) not a lowercase 'L'.

# Add line numbers with custom separator, e.g., ',':
nl -s, -w1 FILE

# Sort lexicographically:
sort FILE

# Sort numerically:
sort -n FILE

# "Transpose" a file, more precisely output content as single row:
tr -d '\n' < FILE

# Get first n (e.g. 10) rows:
head -n 10 FILE

# Get last n (e.g. 10) rows:
tail -n 10 FILE

# Omit the first n (e.g. 10) rows (Get all but the first n rows.):
tail -n +10 FILE

# Omit the last n (e.g. 10) rows (Get all but the last n rows.):
head -n -10 FILE

# head and tail also work character-wise (Beware of the trailing '\n'.):
echo "ABCDE" | tail -c 2
#> E
echo "ABCDE" | tail -c +2
#> BCDE
echo "ABCDE" | head -c 2
#> AB
echo "ABCDE" | head -c -2
#> ABCD

# Add leading zeros:
printf "%03d" 7

# Remove leading zeros:
printf "%d" 007

# Output
######

# Repeat string (e.g. "foo\n") n times (e.g., 10):
printf 'foo\n%.0s' $(seq 1 1 10)

# Concatenation
#############

# Concatenate two files (add FILE2 to the end of FILE1):
cat FILE1 FILE2

# Concatenate two files column-wise:
paste FILE1 FILE2

# Concatenate two files column-wise, with custom separator, e.g., ',':
paste -d, FILE1 FILE2

# Simple Calculations
#################

# Integer operations can be done directly in the shell:
echo $((1+2))

# For floating point operations the "bc" tool can be used:
echo "1.23 * 2" | bc

# Simple Statistics
#################

# Calculate min, mean, median, max, sd, for data in column 3:
awk FILE '{print $3}' | Rscript -e 'd<-scan("stdin", quiet=TRUE); cat(min(d), mean(d), median(d), max(d), sd(d), sep=" ")'

“Rscript” as seen in the last example can generally be used for invoking R.

Another way for invoking R that I found useful is in shell scripts via here files. The following snippet shows a simple script that iterates over all *.raw.data files in a directory, creates a box plot, and stores the boxplot in a *.png file:


#!/bin/bash

for f in DIRECTORY/*.raw.data

do

R --slave --vanilla --no-save --quiet <<R_END
data <- read.table("$f")
png("${f}.png")
boxplot(data)
dev.off()
R_END

done

Of course, this example assumes that the data in the raw data files is already in a format that can be read by R. Something like this can be handy, e.g., if you want to get a quick overview of a bigger number of data files.

Posted in Snippets | Tagged | Leave a comment

Panoramic Photos, “Beautiful British Columbia”, Canada

In this post, I upload some panoramic photos I took in May during a trip to “Beautiful British Columbia”, Canada, which, I think, truly deserves this byname. Even though I just visited a very small part of British Columbia, I really enjoyed the beautiful nature. Unfortunately, I didn’t get around to uploading these photos earlier.

The first photo shows Brohm Lake, which is located some driving minutes north of Squamish.

brohm_lake

Squamish itself is a small town located about an hour drive north of Vancouver. There are many beautiful places around Squamish. The following picture shows the Brandy Wine Falls located about 30 minutes north of Squamish.

brandywine_falls

The next two photos are no panoramas. I add them anyhow as I think these are really beautiful. The photos show the Shannon Falls which are located a few minutes south of Squamish.

The next photo shows a view from Porteau Cove, which is located about 15 minutes south of Squamish.

porteau_cove

In the following panorama, a view from the Tantalus view point at the Sea to the Sky Highway is shown. This view point is located about 15 minutes north of Squamish. Generally, there are many nice viewing points along the Sea to the Sky Highway.

tantalus_view_point

The next panorama does not show nature but the inside of the main building of the Brittania Mining Museum, located about 15 minutes south of Squamish. I added this photo as I think it shows an impressive sight as well.

brittania_mining_museum

The next two pictures were taken on Vancouver Island. The first shows the view from the Holland Point in Victoria. The second shows flowers in Butchard Gardens.

victoria_holland_point

butchart_gardens_1

The last three pictures were taken in Vancouver. The first shows a view from the Ambleside Park. The last two were taken in Lynn Canyon.

vancouver_sea_at_ambleside_park

lynn_canyon_1

lynn_canyon_2

 

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Posted in Misc. | Tagged , | Leave a comment

#Pebble App for #SailfishOS / #Jolla Updated: SkippingStones 0.17

Recently, I have been even more busy than usually. As one consequence, I couldn’t really keep track of announcing improvements in SkippingStones. In this post, I try to summarize the most important changes very briefly and add a link to the latest SkippingStones version. Great thanks also to Uladzislau Vasilyeu (aka vasvlad) for his improvements and pull requests :).

Below, is just an enumeration of the improvements in no particular order:

  • Ability to Manage Watchfaces and Watchapps (Please use this with caution as it happened to me that the watch was set into recovery mode and I had to recover it via the Android app.)
  • Support for some SmartStatus Watchapps (see below)
  • Support for Media Player Controls including Volume via SmartStatus Watchapps
  • Display of Current Weather Icon in SmartStatus with MeeCast (Requires a patch.)
  • Display the Phone Battery Level in SmartStatus
  • Possibility Hangup for PhoneControl
  • Support for “CalendarReminder”
  • Show Name from adressbook for PhoneNumber in incoming call message
  • Fixed problem with non-Latin letters in notification about incoming call

I hope I didn’t forget any important change. Also, I don’t have the time, right now, to add more detailed explanations, e.g., on applying the MeeCast patch etc. For more details etc. see the thread linked below.

Please note that SkippingStones comes without any warranty etc. The new version can be downloaded from: https://github.com/ruedigergad/SkippingStones/blob/master/dist/harbour-skippingstones-0.17-1.armv7hl.rpm?raw=true

So far, I tested the SmartStatus support with these watchapps: http://www.mypebblefaces.com/apps/21105/10414/ http://www.mypebblefaces.com/apps/4805/6490/6508/

If there are no critical errors in this version, I plan to upload it to openrepos as well. However, for this, I need your feedback. ;)

For more information and discussion, please see the following thread: http://talk.maemo.org/showthread.php?t=92695

Posted in My Applications, SkippingStones | Tagged , , , , , | Leave a comment