Answers.Body: <p>We did a lot of work to the dedupe processes in CiviCRM and in 4.7 its a much easier process to de-dupe large databases then it used to be. It was presented by Eileen in this years London CiviCon, worth a look before you get into trying to do it outside of CiviCRM with all the pitfalls that brings with it!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-10T14:09:41.460Z
Answers.LastActivityDate: 2016-11-10T14:09:41.460Z
Answers.UserDisplayName: Parvez Saleh
Body: <p>I'm working on a database with about 600,000 records, of which approximately 200,000 are believed to be duplicates.</p>
  
  <p>Has anyone got experience/advice for doing dedupe at scale?  I imagine that we'll need to export and do some sort of external dedupe.  Has anyone successfully used the <a href="https://github.com/datamade/dedupe">datamade dedupe</a> with Civi?  I know that the "merge" API got some love a couple months ago - will any of that help me?</p>
  
Tags: <duplicate-contacts>
Title: How can I efficiently dedupe a very large number of records?
ThreadText: # How can I efficiently dedupe a very large number of records?
  
  (Asked by: username_0 on 2016-10-24)
  
  I'm working on a database with about 600,000 records, of which approximately 200,000 are believed to be duplicates.
  
  Has anyone got experience/advice for doing dedupe at scale? I imagine that we'll need to export and do some sort of external dedupe. Has anyone successfully used the [datamade dedupe](https://github.com/datamade/dedupe) with Civi? I know that the "merge" API got some love a couple months ago - will any of that help me?
  
  **Tags:** duplicate-contacts
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-11-10. Score: 3
  
  We did a lot of work to the dedupe processes in CiviCRM and in 4.7 its a much easier process to de-dupe large databases then it used to be. It was presented by Eileen in this years London CiviCon, worth a look before you get into trying to do it outside of CiviCRM with all the pitfalls that brings with it!
  
  
  ---
  
  