Speed-up bash scripts that use grep often

The last day I ran into a very awkward situation, but lets start from the beginning:
I wrote a bash script that had to perform several regular expression grep's in several text files.
So far so good, but the script always needed several minutes since the first version. I never wondered about this long execution time, the script runs on a remote server and also uses some mysql searches on another server.
Until some days ago, I wanted to debug a minor error with bash -xv to see all instructions when being executed.
To read the huge output more easily, I logged into the remote server from emacs editor and launched the script and had all the output in my emacs window in a few seconds. Wait, it only needed a few seconds?!
What happened, I rechecked the execution, with exactly the same parameters to the script from my emacs shell and from a ssh terminal.
The same, within emacs it only took some seconds, within the ssh terminal y took minutes.

So what's going on here?
Investigating the processes in the remote server with htop showed that the grep commands consumed almost 100% of the CPU for some time.
But when launched from emacs they did not.
What might be different?
Well, what about the environment?
Emacs might use it's own set of environment, different from a terminal.
Looking at the export definitions of both, I could see that the locale was different, emacs was using the C, whereas the terminal was using the English en_UK.UTF8.
Issuing a export LC_ALL=C in the terminal and then launching the script ... only a few seconds.
So it seems, grep is sensible to the locale of the shell.
Investigating in the web confirmed this, people talk about a speed enhancement of up to 10 times when using the C locale with grep, as it doesn't have to convert anything, it just compares a plain ASCII set of character, which in most cases is enough.
Also, grep is slightly faster when comparing fixed strings if you use the -F flag or call it with fgrep (due to lazyness I always used grep in such cases).

Instructions for speeding up grep

At the beginning of my scripts, I now put these instructions, which will force to use the C locale for all three flavours of grep:
shopt -s expand_aliases
for g in "" e f; do
    alias ${g}grep="LC_ALL=C ${g}grep"  # speed-up grep commands by not considering locale.
done
It does not work to just put export LC_ALL=C at the beginning.

More fortunes for Mint

If you like to see some funny quotes when opening a new shell terminal, you can enable the fortune-mint script that will be launched automatically.
By default, this script only shows non-offensive quotes and only three different types of animals: the moose, the tux and the koala.
But the cowsay package provides a whole bunch of pictures that can be used.
Cow files in /usr/share/cowsay/cows:
apt beavis.zen bong bud-frogs bunny calvin cheese cock cower daemon default
dragon dragon-and-cow duck elephant elephant-in-snake eyes flaming-sheep
ghostbusters gnu head-in hellokitty kiss kitty koala kosh luke-koala
mech-and-cow meow milk moofasa moose mutilated pony pony-smaller ren sheep
skeleton snowman sodomized-sheep stegosaurus stimpy suse three-eyes turkey
turtle tux unipony unipony-smaller vader vader-koala www

The following patch will enable all these that you have installed in your system, as well as all different eye and tongue styles. If you don't like offensive quotes, you'll have to take off the -a parameter from the fortune command in the last line of the patch below.

Install instructions

Paste the following into a textfile, for example mint-fortune.patch
4c4
<     RANGE=3
---
>     declare -a cows=( $(cowsay -l | grep -v "Cow files in") )
6,17c6,7
<     let "number %= $RANGE"
<     case $number in
<         0)
<             cow="moose"
<             ;;
<         1)
<             cow="tux"
<             ;;
<         2)
<             cow="koala"
<             ;; 
<     esac
---
>     let "number %= ${#cows[*]}"
>     cow=${cows[$number]}
19c9
<     RANGE=2
---
>     declare -a says=( say think )
21,30c11,19
<     let "number %= $RANGE"
<     case $number in
<         0)
<             command="/usr/games/cowsay"
<             ;;
<         1)
<             command="/usr/games/cowthink"
<             ;;
<     esac
<     /usr/games/fortune | $command -f $cow
---
>     let "number %= ${#says[*]}"
>     command="/usr/games/cow${says[$number]}"
> 
>     declare -a states=( "" -b -d -g -p -s -t -w -y )
>     number=$RANDOM
>     let "number %= ${#states[*]}"
>     state=${states[$number]}
> 
>     /usr/games/fortune -a | $command $state -f $cow

Then apply this patch to the original script (a backup will be created) and enable the fortune-teller:
sudo patch -b /usr/bin/mint-fortune mint-fortune.patch
gconftool-2 --set /desktop/linuxmint/terminal/show_fortunes true --type bool

Provide your own cows

You can create your own cow files, just have a look at the manual of the cowsay command. Or just copy the ones I prepared from here into any folder you want.
You could just copy them into the system folder /usr/share/cowsay/cows, but if you want to keep them separated, you can instruct cowsay to look for you ones too (for example in your folder ~/Documents/cows:
mycows=~/Documents/cows ; sudo sed -i.orig 's|/usr/bin/mint-fortune|export COWPATH=/usr/share/cowsay/cows:'$mycows'\n\0|' /etc/bash.bashrc

Example of a Star Wars scene as cow

 _____________
< Hello world >
 -------------
          \
           \                .==.
            \              ()oo()-.
             \  .---.       ;--; /
              .'_:___". _..'.  __'.
              |__ --==|'-''' \'...;
              [  ]  :[|       |---\
              |__| I=[|     .'    '.
              / / ____|     :       '._
             |-/.____.'      | :       :
            /___\ /___\      '-'._----' 

SSH key handling

If you have to work with servers, especially with Linux ones, sooner or later you'll have a confrontation with SSH.

I'll give here some tips and tricks I used so far.

Force user and port for certain servers

When you try to login into a server, by default ssh uses your username and port 22 by default.
But maybe you need to use always another user name or another port.
You could specify this always as parameters for ssh:
ssh -p 22022 [notme@]server-ip

But it is much easier to configure your ssh client to these by default.
Therefore, you just add the following to your ~/.ssh/config file:
host server-ip another-server-name
Port 22022
User notme

host *
User root
You can put here as many hosts with different parameters as you want, '*' is also supported for creating regular expressions.

Change order of authentication methods

There's a another very useful parameter that you might want to add to your server configurations:
host *
User root
PreferredAuthentications publickey,password

This would only allow to use keys or passwords and prevents to use other methods which might not work in your setup and slow down connection attempts.
Put this whenever you notice that it take several seconds to log into a server.

Copy your own key to server

This used to be the first step I do, whenever I access a certain server several times.
So I don't have to give the password each time I access the server.
# Copy my machines public key to the server (will prompt for password):
ssh-copy-id [user@]server-ip

# Unfortunately, ssh-copy-id only works with SSH port 22, so if you have to specify another
# one, you might use this instruction:
ssh-add -L | ssh -p22022 [user@]server-ip "umask 077; test -d ~/.ssh || mkdir ~/.ssh ; cat >> ~/.ssh/authorized_keys"

# Verify the granted access (now without password prompt):
ssh [user@]server-ip
# or when copying
scp afile.txt server-ip:test/ 

Fix non-working ssh public keys

Sometimes, the sshd server doesn't accept a previous copied ssh-key (ssh-copy-id). In that case, make sure you have the following configuration in /etc/ssh/sshd_config
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# If it still doesn't work apply the correct permissions to folders and files:
# chmod go-w ~
# chmod 700 ~/.ssh
# chmod 600 ~/.ssh/authorized_keys

# If it still doesn't work, then disable the permissions checking:
StrictModes no
Restart the ssh daemon again after applying this new configuration.

Debugging and sorting out further problems

The permissions of files and folders is crucial. You can get debugging information from both the client and server. if you think you have set it up correctly, yet still get asked for the password, try starting the server with debugging output to the terminal. /usr/sbin/sshd -d
To connect and send information to the client terminal ssh -v ( or -vv) username@host's

Remove authorized keys

If you have hundreds of keys in your machines ~/.ssh/authorized_keys and you're to lasy to edit that by hand, these one-line shell commands maybe handy.
Just use sed to rip out anything which doesn’t match the regex pattern (for example machine names, part of the hash, whatever:
# rip out anything which doesn’t match the regex pattern
sed ‘/your host name/ ! D’ -i.old ~/.ssh/authorized_keys

# or something more complicated
sed ‘/\(host1\|host2\)/ ! D’ -i.old ~/.ssh/authorized_keys

# with many patterns it is easier with this command
cp ~/.ssh/authorized_keys{,.old} && for p in pat1 pat2 pat2 ; do sed '/$pat1/ ! D' ~/.ssh/authorized_keys ; done

Just the ones specified will be maintained, a backup file will be created.
If you want to do the opposite, remove some specific keys and let the rest untouched:
sed ‘/\(host1\|host2\)/ D’ -i.old ~/.ssh/authorized_keys

Music Player Clementine

A few days ago, I discovered another music player which I'm using now daily: Clementine.

Features I like:
  • It's quick and stable.
  • Mood graphics of songs.
  • Dynamic random playlists.
  • Discover song information, covers and lyrics from variety of sites.
  • Cover manager to retrieve from even more sources.
  • Extensive artist information gathered from various sources.
  • Tons of visualization algorithms, random changes.
  • It integrates well with Cinnamons sound gadget.
Unfortunalty, the handling of external devices isn't perfect yet.
At least, I was not able to add songs from my external harddisks to the random playlist, I had to create a new playlist for that purpose.

Install instructions

Use Y-PPA-Manager and search for clementine package to obtain list of repositories or use the following shell commands:

sudo apt-add-repository ppa:me-davidsansome/clementine
sudo apt-get update
sudo apt-get install clementine

# You need to install the developer version for version 1.1, which features mood bars)
sudo apt-add-repository ppa:me-davidsansome/clementine-dev