title: Getting a hash value of a binary file with python by md5, trying to weed out duplicates
body: <p><em>The issue:</em> file name will result in a different hash value even if content of file is the same</p>
  
  <hr>
  
  <p>Found the issue where the bug was in the MD5 buffer. We needed to declare a new hasher for each file. It was updating, the stream from the previous hash before. Uploaded a working version, will clear out all your duplicate files. </p>
  
  <p>I'm trying to find all duplicate files under my directory, and delete all duplicate files (by their hash value) and leave one unique file left over. </p>
  
  <p>I'm opening up files from a directory with <code>rb</code> and using md5 to get the hexdigest of each file but I find that files with a different name will hash out different values even though the content of the file is the same. This is true when trying to hash sound files, maybe because there is ID3 headers that are different based on the file name of the sound clip. </p>
  
  <p>Is there a way to verify the integrity of each file's content without consideration of the file name? </p>
  
  <p>Here I have posted my little personal script <a href="http://pastebin.com/YZYXKMpt" rel="nofollow">http://pastebin.com/YZYXKMpt</a></p>
  
  <p><strong>Results:</strong> Same hash values. Different :(</p>
  
  <pre><code>&gt;&gt;&gt; md5("C:\\Users\\c\\Documents\\Test\\06 Liar Liar.mp3")
  'f9c50701d728c87cd959e8b7c6e7ebb2'
  &gt;&gt;&gt; md5("C:\\Users\\c\\Documents\\Test\\06 Liar Liar - Copy.mp3")
  'f9c50701d728c87cd959e8b7c6e7ebb2'
  
  &gt;&gt;&gt; sha1("C:\\Users\\c\\Documents\\Test\\06 Liar Liar - Copy.mp3")
  '68dcad808ab68e6b1f8610d21e92944ceb45086f'
  &gt;&gt;&gt; sha1("C:\\Users\\c\\Documents\\Test\\06 Liar Liar.mp3")
  '68dcad808ab68e6b1f8610d21e92944ceb45086f''
  
  {'C:\\Users\\c\\Documents\\Test\\06 Liar Liar.mp3': 'c7c8c7f9791804d3e1951bebb5d1494025da458c', 
  'C:\\Users\\c\\Documents\\Test\\06 Liar Liar - Copy.mp3': '64097f07a8c40488a74fb98a4b9e83d7e7885f6c', 
  'C:\\Users\\c\\Documents\\Test\\06 Liar Liar - Copy (2).mp3': '68dcad808ab68e6b1f8610d21e92944ceb45086f'}
  </code></pre>