title: Finding and Listing Duplicate Words in a Plain Text file
body: <p>I have a rather large file that I am trying to make sense of.
  I generated a list of my entire directory structure that contains a lot of files using the du -ah command.
  The result basically lists all the folders under a specific folder and the consequent files inside the folder in plain text format.</p>
  
  <p>eg:</p>
  
  <pre><code>4.0G    ./REEL_02/SCANS/200113/001/Promise Pegasus/BMB 10/RED EPIC DATA/R3D/18-09-12/CAM B/B119_0918NO/B119_0918NO.RDM/B119_C004_0918XJ.RDC/B119_C004_0918XJ_003.R3D
  3.1G    ./REEL_02/SCANS/200113/001/Promise Pegasus/BMB 10/RED EPIC DATA/R3D/18-09-12/CAM B/B119_0918NO/B119_0918NO.RDM/B119_C004_0918XJ.RDC/B119_C004_0918XJ_004.R3D
  15G ./REEL_02/SCANS/200113/001/Promise Pegasus/BMB 10/RED EPIC DATA/R3D/18-09-12/CAM B/B119_0918NO/B119_0918NO.RDM/B119_C004_0918XJ.RDC
  </code></pre>
  
  <p>Is there any command that I can run or utility that I can use that will help me identify if there is more than one record of the same filename <strong>(usually the last 16 characters in each line + extension)</strong> and if such duplicate entries exist, to write out the entire path (full line) to a different text file so i can find and move out duplicate files from my NAS, using a script or something.</p>
  
  <p>Please let me know as this is incredibly stressful to do when the plaintext file itself is 5.2Mb :)</p>