The idea is to use a two stage process:<p>1. take a sha1 hash of the last 4KB of each file.<p>2. for any equal hashes, compare the <i>whole</i> file.<p>With this method you should be able to skip reading many large files in their entirety.