Features

• Very fast (written in pure C, in many cases faster than rdfind, and always magnitudes faster than fdupes).
• Output of both a ready to use script to handle finds and a easy-to-parse logfile.
• Tries to minimize I/O as much as possible (focus on CPU-usage).
• Finds duplictes, nonstripped binaries, files with same basenames (nameclusters),
  
  empty files/directories, old tempdata, strange filenames
• Displays finds in realtime.
• Safely abortable at any time (will write log & script).
• Colorful output (can be disabled via -B).
• ...and some more goodies to find for you :-)

Dependencies

Only requirement is a POSIX-compilant Operating System, with a glibc version > ancient
Till now it has been succesfully tested on Linux 32/64bit OS.

Install

## DOWNLOAD

# Git way
git clone https://github.com/sahib/rmlint.git 

# OR

# Zip way
wget --no-check-certificate http://github.com/sahib/rmlint/zipball/master -O rmlint.zip
unzip rmlint.zip

# OR

# Tar way
wget --no-check-certificate http://github.com/sahib/rmlint/tarball/master -O rmlint.tar
tar xfv rmlint.tar

# compile it

cd <name_of_the_dir>
./configure
make -j 4 
sudo make install 

Usage

Please see 'man rmlint' or 'rmlint -h' for detailed information.
Note that rmlint will *never* delete anything without your confirmation.

License

rmlint is licensed under the terms of the GPLv3

Author

Christopher <Sahib> Pahl (sahib@online.de)
Send all suggestions / bugreports / patches to this address - Thanks!
See also my github-page for more info on me.

Disclaimer

rmlint was written carefully in the hope to be useful, but I don't give any warranty that it won't eat your data!
If this happens, and if it was due to a bug in rmlint, please
make a bugreport, so offers won't suffer from it!


A note about false positives:

Short: False Positves are actuallly possible, but very, very, very unlikely.
They would need to have the same size, fingerprint and checksums to be marked as twins.
md5 is not perfect, but the probability of getting false positves on a normal set of data is the same as lim(1/x) : x -> +inf = 0 + h; where h ~ 0

But isn't there a solution to be 100% sure? Yes, there is. It's the "--paranoid/-p" option. It does a
true byte-by-byte comparasion of each(!) file. Be warned, because it's incredibly slow.

What it does internally (Algorithm):

The algorithm tries to mimize IO as far as possible, thus focusing on CPU usage:

• 1) Go through all directories and catch all files conformig to regexpattern / dirpattern / hiddenstatus
  
  
• 2) lint other than duplicates get detected here on the fly (like nonstripped binaries - every file is checked)
  
  
• 3) the rest of the list (all files without files from 2)) gets sorted by their filesize
  
  
• 4) elements with a unique filesize gets kicked out (because they can't have a twin)
  
  
• 5) list gets divided isn sublist, each size one sublist
  
  
• 6) each sublist gets sort by inode (to speed up reading from HD)
  
  
• 1) Each group is processed seperately:
  
  
  • 7.1) if the size of group exceeds a certain limit then it's processed on an own thread
    
    
  • 7.2) else the group gets processed within the main thread
    
    

• 8) Processing: For each file of a group..
  
  

  • 8.1) A short fingerprint from the start/end + some bytes in the middle of the file is read and stored
    
    
  • 8.2) Nonmatching files get kicked out, if the group consists of 1 elem or less, rmlint forgets about it
    
    
  • 8.3) md5sums are calculated for the rest of the group
    
    (only the part of the file that hasn't been read, is used fo md5sum calculation!)
    
    
  • 8.4) if the groupssize exceeds a certain limit, the group gets splitted into several equalsized subgroups
    
    
  • 8.4.1) The whole file is read blockwise, while other threads have wait (so no useless jumping is done)
    
    
  • 8.4.2) After a block is read (blocksize is about 2MB) md5 is updated,
    
    while at the same time another thread is reading, back 8.3.1)
    
    
  
  
  
  • 8.5) md5sums, filesize, fingerprint and bytes in the middle get checked each other (to double check and prevent false positives)
    
    
  • 8.6) log/handle result to script / log / screen (let other threads wait for this short time, so no chaos is created)
    
    
• 9) Do 8) for every group, and print statistics when finished.
  
  
  
  
  

(Pseudo-)Speed comparasion:

• Tests were made on a AMD Phenom(tm) 9950 Quad-Core Processor
• 5 runs each were made, and the average of run #2,3,4,5 were taken
• This is not a real benchmark: rdfind is often faster on singlecoremachines!




	rdfind	fdupes	rmlint	notes:
88GB Documents:	1,430s	8,137s	0.656s	rmlint CPU usage was 310%, rdfind's 99%
2,2GB of Source:	12.030s	30.552s	1.641s	rdfind was faster on the first run.
50GB of Music:	0.089s	1:54min	0.097s	Dir did not contain any twins.

get the developement version on GitHub : sahib/rmlint