Duplicate file cleanup

CloneSpy (Windows)

CloneSpy can help you free up hard drive space by detecting and removing duplicate files. Duplicate files have exactly the same contents regardless of their name, date, time and location. Also, CloneSpy is able to find files that are not exactly identical, but have the same file name, or the file size differs only a bit.

FSlint (Linux)

FSlint is an utility to fix problems with filesystems' data, like duplicate files
is a toolkit to clean filesystem

fdupes (Linux)

FDupes uses md5sums and then a byte by byte comparison to find duplicate files within a set of directories. It has several useful options including recursion. This is the fastest one.

sudo aptitude install fdupes

You can instruct fdupes to delete duplicate files automatically, but you can't be sure it will delete always from the system folder.
Therefore, you could execute the following command:

fdupes -r a/ b/ | grep -o "^b/.*" | xargs -d '\n' rm ; find b/ -empty -delete

Linux shell commands

In an Ubuntu forum, they worked out a pipe of shell commands, to generate the same output as fdupes. So you don't have to install any software, but it's also much more slower (see benchmarks below).

find . ! -empty -type f -printf "%s " -exec ls -dQ {} \; | sort -n | uniq -D -w 1 | \
cut -d" " -f2- | \
xargs md5sum | sort | \
uniq -w32 -d --all-repeated=separate | \
cut -c35-

Benchmarks

722 groups
994 duplicate files
6205 files

FSlint 10 minutes
fdupes 5,5 minutes
Piped commands 13 minutes

Hard links

If you don't want to remove duplicates, but save memory you can use hardlink (or fdupes) on Linux which detects multiple copies of the same file and replaces them with hardlinks.
For example in my case:

Files:    5935
Linked:   991 files
Compared: 4022 files
Saved:    4.87 GiB
Duration: 12.6 minutes

Extension collections for Firefox and Thunderbird

On Mozilla they started a new kind of extension sharing, so called Collections.
You can create your own set of useful extensions and share them with the whole world.

So here I'll share two collections with you, one for Firefox, the other for Thunderbird.
These collections are the extensions that I've installed and use on a daily basis.

• Extension collection for Firefox


• Extension collection for Thunderbird

Video Edition for Linux

You have some nice vacation video in your Camcorder and want to edit it on your Linux system.
It seems, that Windows user are more luckily, lots of video edition software exists (for example ULead), but Linux users also can edit their videos with free software.

In this post, I'll comment about three applications, which are powerful and easy to use.

Kino

Kino is a non-linear digital video editor which supports many basic video editing and assembling tasks.
But its main feature is the import from raw AVI and DV files, as well as capture footage from digital camcorders using the raw1394 and dv1394 libraries, and export to camcorders using the ieee1394 or video1394 libraries.
During import, it detects scene changes (when you stopped grabbing) and can split the scenes into separate files.

For me, its the application which works best when passing my videos from the digital camera onto the hard drive, without frame dropping.

Install instructions

sudo aptitude install kino

or click here to install it from the browser.

Troubleshooting

A normal user doesn't have privileges to the RAW 1394 device, which is used to transfer data from the camcorder to the computer. You would have to launch Kino as superuser or execute the following commands:

sudo modprobe raw1394
sudo chmod a+rw /dev/raw1394

To make these changes permanent:

sudo echo "install raw1394 /sbin/modprobe --ignore-install raw1394 $CMDLINE_OPTS ; \
                sleep 0.5 ; \
                /bin/chmod a+rw /dev/raw1394" > /etc/modprobe.d/raw1394.modprobe

OpenShot

OpenShot is a video editor for GNU/Linux, being simple but yet powerful. It has all functionality one would need for day-to-day video cutting and assemblage.

Its interface supports themes, is very eye-candy and integrates well with Gnome.
You can see a list of features on its website.

Install instructions

sudo add-apt-repository ppa:jonoomph/openshot-edge
sudo apt-get update
sudo apt-get install openshot openshot-docs

or see instructions here if you don't use Ubuntu Karmic.

Kdenlive

Kdenlive is an intuitive and powerful multi-track video editor, including most recent video technologies.
It has tons of video and audio effects and video snippets can be arranged visually on the several tracks.
The resulting video can also be exported to DV devices, or written to a DVD with chapters and a simple menu.

Where Kino's importing feature is more powerful, Kdenlive is so much better when editing and creating the "final cut".

Install instructions

sudo aptitude -y install kdenlive

or click here to install it from the browser.

DVD Authoring

You should install the following additional programs, so you can export the video from Kdenlive to a DVD folder which can be burned to DVD directly.

sudo aptitude install ffmpeg lame mjpegtools dvdauthor

Thunderbird 3.0 and Lightning

Similar to my previous post about Firefox 3.5, you can also go and use Thunderbird 3.0 already.

Even if its codename is Shredder and some of your extensions will stop working, I highly recommend this beta version,

• especially if you access IMAP mail accounts, because the access is so much faster now.
• the automated account creator which guesses the servers ports and settings is also a great invention I haven't seen in any mail client so far.
  

But not everything is gold, as mentioned before, most of your extensions will not work yet (see below how to get Lightning work, the most important one). I also don't like so much the cosmetic change in the layout which puts information like tags, and command buttons above the message content. This eats up to much space, especially for small screens like laptops.

Install instructions

sudo -v

echo "deb http://ppa.launchpad.net/ubuntu-mozilla-daily/ppa/ubuntu jaunty main" > /tmp/MozillaDaily.list
sudo mv /tmp/MozillaDaily.list /etc/apt/sources.list.d
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 247510BE
sudo apt-get update
sudo aptitude install thunderbird-3.0 thunderbird-3.0-gnome-support

# Create a new entry in gnomes alternatives system and set Thunderbird 3.0 to be the preferred choice.
sudo update-alternatives --verbose --install /usr/bin/thunderbird thunderbird /usr/bin/$(readlink /usr/bin/thunderbird) 60
sudo update-alternatives --verbose --install /usr/bin//usr/bin/thunderbird thunderbird /usr/bin/thunderbird-3.0 70
sudo update-alternatives --verbose --auto thunderbird
# See current configuration, should result in:
## thunderbird - status is auto.
##  link currently points to /usr/bin/thunderbird-3.0
## /usr/bin/../lib/thunderbird/thunderbird - priority 60
## /usr/bin/thunderbird-3.0 - priority 70
## Current `best' version is /usr/bin/thunderbird-3.0.
sudo update-alternatives --verbose --display thunderbird
# If you get into any trouble and want to use your previous Thunderbird again, just change the selection with
## sudo update-alternatives --config thunderbird

Get Lightning working again

The latest oficial release of Lightning will not work with the Thunderbird 3.0 pre release, so you'll have to install the latest nightly build.

wget http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-1.9.1/linux-xpi/lightning.xpi
# If you want to connect to Google Calendar:
wget http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-1.9.1/linux-xpi/gdata-provider.xpi

As Thunderbird 3.0 pre gets updated, from time to time, Lightning could stop working again.
Just download again the latest nightly build and install it again.
To make things easier, someone created an extensions for updating to nightly builds.

Firefox 3.5 is out

They released the latest version of this magnificent web browser.

Go here to see real-time download statistics and a world map where Firefox 3.5 is downloaded.

Install instructions

sudo -v

echo "deb http://ppa.launchpad.net/ubuntu-mozilla-daily/ppa/ubuntu jaunty main" > /tmp/MozillaDaily.list
sudo mv /tmp/MozillaDaily.list /etc/apt/sources.list.d
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 247510BE
sudo apt-get update
sudo aptitude install firefox-3.5 firefox-3.5-gnome-support latex-xft-fonts

# Create a new entry in gnomes alternatives system and set Firefox 3.5 to be the preferred choice.
sudo update-alternatives --verbose --install /usr/bin/firefox firefox /usr/bin/$(readlink /usr/bin/firefox) 60
sudo update-alternatives --verbose --install /usr/bin/firefox firefox /usr/bin/firefox-3.5 70
sudo update-alternatives --verbose --auto firefox
# If you get into any trouble and want to use your previous Firefox again, just change the selection with
# sudo update-alternatives --config firefox

Note

Some instructions come from this post.