Speed-up bash scripts that use grep often

The last day I ran into a very awkward situation, but lets start from the beginning:
I wrote a bash script that had to perform several regular expression grep's in several text files.
So far so good, but the script always needed several minutes since the first version. I never wondered about this long execution time, the script runs on a remote server and also uses some mysql searches on another server.
Until some days ago, I wanted to debug a minor error with bash -xv to see all instructions when being executed.
To read the huge output more easily, I logged into the remote server from emacs editor and launched the script and had all the output in my emacs window in a few seconds. Wait, it only needed a few seconds?!
What happened, I rechecked the execution, with exactly the same parameters to the script from my emacs shell and from a ssh terminal.
The same, within emacs it only took some seconds, within the ssh terminal y took minutes.

So what's going on here?
Investigating the processes in the remote server with htop showed that the grep commands consumed almost 100% of the CPU for some time.
But when launched from emacs they did not.
What might be different?
Well, what about the environment?
Emacs might use it's own set of environment, different from a terminal.
Looking at the export definitions of both, I could see that the locale was different, emacs was using the C, whereas the terminal was using the English en_UK.UTF8.
Issuing a export LC_ALL=C in the terminal and then launching the script ... only a few seconds.
So it seems, grep is sensible to the locale of the shell.
Investigating in the web confirmed this, people talk about a speed enhancement of up to 10 times when using the C locale with grep, as it doesn't have to convert anything, it just compares a plain ASCII set of character, which in most cases is enough.
Also, grep is slightly faster when comparing fixed strings if you use the -F flag or call it with fgrep (due to lazyness I always used grep in such cases).

Instructions for speeding up grep

At the beginning of my scripts, I now put these instructions, which will force to use the C locale for all three flavours of grep:

shopt -s expand_aliases
for g in "" e f; do
    alias ${g}grep="LC_ALL=C ${g}grep"  # speed-up grep commands by not considering locale.
done

It does not work to just put export LC_ALL=C at the beginning.

Access Subversion from Emacs

Emacs is a feature-rich text editor, with perhaps, more editing commands than any other editor or word processor

There exists lots of documentation in Internet about this great editor, for example the following links are very useful:

• Wiki about Emacs
  (here you can find lots of material, extensions)


• Oficial web page
  

Integration with Subversion

There exist several plugins for the integration of Emacs with Subversion
, but the best one is Svn Status Mode. It's work is based on previous plugins, like vc-svn and dsvn.

Installation

You can get the very latest version with any of the following commands:

1. sudo svn export http://svn.collab.net/repos/svn/trunk/contrib/client-side/emacs/psvn.el /usr/share/emacs/site-lisp/psvn.el




2. sudo wget -O /usr/share/emacs/site-lisp/psvn.el http://www.xsteve.at/prg/emacs/psvn.el




3. sudo wget -O /usr/share/emacs/site-lisp/psvn.el http://svn.collab.net/repos/svn/trunk/contrib/client-side/emacs/psvn.el

Add the following line to your ~/.emacs

(require 'psvn)

How to use

From now on, you find a new menu entry Tools-->SVN Status in the XEmacs.

• You can also launch the command Alt-x svn-status directory .
  


• It opens a new buffer *svn-status* where you can see all files controlled by Subversion, with flags that indicate there state.


• Inside the XEmacs you have a menu which gives you access to all kind of commands for commiting and more, if not, just have a look at the mentioned Wiki or issue the command Ctrl-h m.

subcommander

Recently, I discovered another graphical client for the subversion source control system.

Subcommander is a cross platform (MacOSX, Windows & Linux) client with diff and merge tool and integrates very well into Ubuntu, and supports much better subversions trunk, branches and tags layout.


Compared to eSvn it can browse and work directly with repositories, and has all features from kdesvn.
See also my previously post about subversion GUIs.

It can be installed from Synaptic, aptitude or apt-get, but unfortunately doesn't create an entry in the applications menu.

Gantt Charting

A Gantt chart is a popular type of bar chart that illustrates a project schedule. Gantt charts illustrate the start and finish dates of the terminal elements and summary elements of a project.

I don't like Gantt charts much, which you'll understand if you work with Agile Development methodologies like SCRUM.
But these kind of diagrams are still heavily used in lots of projects, so sooner or later you'll need a program to handle them.

Windows

There's the mother of all: MS Project.
What should I say, as many of MS Office programs, its heavy, slow, and error-phrone. And it's not free and doesn't run on Linux.

Ubuntu

For Ubuntu there are several free applications available, and some of them can handle MS Project files, which is very important if your boss only handles this program.

Read my special article about this kind of applications, their dis-/advantages, special installation instructions aso.

Visual Diff and Merge tools

Working with source controls systems implies to view differences between versions, merge changes.
Or you just want to see quickly a difference between two files or folders.

On Linux you could stick to good and old tool diff, but it's a command line tool and it gets very confusing when viewing differences between big files.


It's much better to use visual tools for these task, moreover they integrate well with source control tools like explained in one of my previous topics.

For Ubuntu there is meld.

For Windows I recommend WinMerge.