title: Removing duplicates on very large datasets
body: <p>I'm working on a 13.9 GB csv file that contains around 16 million rows and 85 columns. I know there are potentially a few hundred thousand rows that are duplicates. I ran this code to remove them</p>
  
  <pre><code>import pandas
  
  concatDf=pandas.read_csv("C:\\OUT\\Concat EPC3.csv")
  nodupl=concatDf.drop_duplicates()
  nodupl.to_csv("C:\\OUT\\Concat EPC3- NoDupl.csv",index=0)
  low_memory=False  
  </code></pre>
  
  <p>However this runs me into a MemoryError. My ram is 16gb and can't go any higher. Is there a more efficient way of removing duplicates that perhaps does it chunks without me having to break up the csv file into smaller files?</p>