Answers.Body: <p>The import code's use of dedupe rules as of this writing (Civi 5.1) is...interesting.</p>
  
  <p>The rules, which are defined in <code>CRM_Contact_Import_Parser_Contact::getPossibleContactMatches</code> involve short-circuiting the dedupe matching if the external identifier is a match.  After that, it will ALSO run the unsupervised dedupe - and from there, run your dedupe as well.</p>
  
  <p>While it'd be great to fix this (it's <a href="https://github.com/civicrm/civicrm-core/blob/master/CRM/Contact/Import/Parser/Contact.php#L2053" rel="noreferrer">happening here</a>) realistically it will take someone putting in resources (coding time or hiring a coder) to resolve this.</p>
  
  <p>For now, the workaround is to modify your unsupervised rule.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-15T09:47:40.060Z
Answers.Text: Should I open an issue at jira to report this behavior?
Answers.UserDisplayName: Albert V
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-18T05:00:44.600Z
Answers.Text: Still true as of Civi 5.12.1. Note the justification for checking the external ID: it's a unique key for contacts, so if the new record matches on external ID, the only two options are to merge/overwrite the new record onto the matching existing record, or to reject the new record altogether. The code chooses to reject the new record if other dedupe criteria would lead to two records having the same external ID.
Answers.UserDisplayName: Noah Miller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-09T16:04:29.993Z
Answers.LastActivityDate: 2018-05-09T16:32:32.413Z
Answers.LastEditDate: 2018-05-09T16:32:32.413Z
Answers.UserDisplayName: Jon G - Megaphone Tech
Answers.Body: <p>Still having this issue in January 2021, with version 5.32.1 <em><strong>But the suggested workaround works fine.</strong></em></p>
  <p>To summarize the long-winded explanation below: <strong>When trying to import contacts using a dedupe rule of External ID Only, it will not work as expected.</strong> <em><strong>An effective workaround is to temporarily replace the unsupervised rule (Email only) with the desired rule (External Id Only).</strong></em> Things work as expected, and afterward, unsupervised rule can be changed back to (Email only).</p>
  <p>Our problem - an organization has been keeping a flat file in a spreadsheet for years, with columns for various meetings, volunteer dates, etc.
  So trying to import 2400 rows of data with about 50 columns of sparse data. Individual constituents can appear multiple times in the rows, and rows are in order by when the data was collected, making information in a particular row more recent (accurate) than in prior rows. The desired total information import for a contact would then be the latest information in each column, and where the latest row is empty for a given column, then the data in the next most recent row is used, and so on.</p>
  <p>So we have made up a unique External ID for each row that effectively uniquely identifies each individual. Since we already have contacts being generated by program registrations through CiviEvent, we want to have the info in the system (e.g. email address - this latest piece of data is presumed to be more accurate than earlier transactions with this person) be kept, but have any blank fields filled in from the most recent row for that contact, (i.e. the &quot;fill&quot; option in contact import), etc. until there are no more rows for that contact.</p>
  <p>Our import spreadsheet is in ascending order of External ID, and then secondarily in descending order of the order that was in the original spreadsheet. Thus, for e.g. External ID 24, there might be three rows (e.g. 31-33) of data for that individual, and so the import spreadsheet has ID 24 in three rows, which in the import spreadsheet are in the order of 33, 32, 31. That is, for a given External ID, the most recent data is encountered in the import spreadsheet first, then the second most recent second, etc.</p>
  <p>So we set up a dedupe rule &quot;External ID only&quot; and use it with the &quot;fill&quot; option. We presumed that the data in the system (where we had manually added the correct external id) would prevail, augmented by any columns in 33, then 32, then 31, where only the first (latest) information would be retained.</p>
  <p><strong>We were then surprised that the import didn't work, and only a few of the contacts were being imported with a test file of the first 100 rows</strong>. Finding this issue here on Stack Exchange, we realized we were being affected by something similar (or the same - we haven't looked at the code as suggested).</p>
  <p><strong>We used the workaround suggested; changed the unsupervised rule</strong> (email only) to general, created a new rule (External ID only) and made it the unsupervised rule, then did our import. It works exactly as expected. After the big import is completed, we will change the unsupervised rule back to email only (so that people signing up for events won't all result in duplicate contacts, etc.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-13T19:09:09.773Z
Answers.LastActivityDate: 2021-01-13T20:56:59.040Z
Answers.LastEditDate: 2021-01-13T20:56:59.040Z
Answers.UserDisplayName: Rick Cunningham
Answers.Body: <p>Possibly related is that when Skip is selected (the default option) Contact ID is not shown as a match possibility.   Contact ID does show when Fill or Update or No Dedupe is selected.    This seems like a bug.    (When was Fill added to the options?   did that cause regression?)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T16:57:03.697Z
Answers.LastActivityDate: 2018-06-19T16:57:03.697Z
Answers.UserDisplayName: Dave T
Body: <p>Recently I have encountered a problem related to the import of contacts and has to do with the dedupe rules.</p>
  
  <p>When you import a list of contacts through a CSV file and select a dedupe rule, the CRM seems to ignore that rule and go directly to use the 'Unsupervised' rule.</p>
  
  <p>For example, if there is a user with mail xxxxx@eeee.com and you use a 'General' rule with the Weight Threshold and the Weight of 10 in email but in the 'Unsupervised' rule the weight is 10 and the Weight Threshold is 15, the mail would be introduced in the CRM even if the rule that was chosen was the 'General'.</p>
  
  <p>This behavior also happens in dmaster.</p>
  
  <p>Is there any reason to prioritize the 'Unsupervised' rule to the rule you have chosen to perform the import?</p>
  
  <p>EDIT</p>
  
  <p>After more tests I have been able to conclude the following:</p>
  
  <p>Importing data: Name, Last name and email and selecting the skip option.</p>
  
  <p>Unsupervised rules:</p>
  
  <pre><code>A) Name (10) Last name (10) Treshold (30)
  B) Email (10) Treshold (10)
  </code></pre>
  
  <p>General rules:</p>
  
  <pre><code>C) Name (10) Last name (10) Treshold (20)
  D) Name (10) Last name (10) Mail (5) Treshold (25)
  </code></pre>
  
  <p>Then:</p>
  
  <p>A + C = duplicate entry. If the general rule would have been applied correctly should not been duplicated, just skipped.</p>
  
  <p>A + D = duplicate entry.</p>
  
  <p>B + C = skip the entry (works fine)</p>
  
  <p>B + D = skip the entry (works fine)</p>
  
  <p>So as Jon G - Megaphone Tech said, the unsupervised dedupe rule always run, and from there, run your dedupe as well. </p>
  
  <p>So the import first check the unsupervised rule and if the Treshold is greater than the weight of the fields always will import the contact independently the general rule you select.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-09T20:04:44.620Z
Comments.Text: Can you confirm how you are sure which rule is ends up using? Note that if you have a Contact with more than one of the same email, then each instance of that email is counted towards the Threshold. Hence we recommend something like First 10, Last 10, Email 1, Threshold 21 and not F 10, L10, E10, with T of 30, as the latter will count as a match for a contact that has the same email 3 times (it happens!) even if F and L do not match.
Comments.UserDisplayName: petednz - fuzion
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-10T10:00:45.137Z
Comments.Text: I will do more tests to confirm the problem and being carefull with the weight of each field. When I do this tests i'll edit the main posts and show the results.
Comments.UserDisplayName: Albert V
Tags: <import><duplicate-contacts>
Title: Import Contacts with dedupe rules
ThreadText: # Import Contacts with dedupe rules
  
  (Asked by: username_0 on 2018-05-09)
  
  Recently I have encountered a problem related to the import of contacts and has to do with the dedupe rules.
  When you import a list of contacts through a CSV file and select a dedupe rule, the CRM seems to ignore that rule and go directly to use the 'Unsupervised' rule.
  For example, if there is a user with mail xxxxx@eeee.com and you use a 'General' rule with the Weight Threshold and the Weight of 10 in email but in the 'Unsupervised' rule the weight is 10 and the Weight Threshold is 15, the mail would be introduced in the CRM even if the rule that was chosen was the 'General'.
  This behavior also happens in dmaster.
  Is there any reason to prioritize the 'Unsupervised' rule to the rule you have chosen to perform the import?
  EDIT
  After more tests I have been able to conclude the following:
  Importing data: Name, Last name and email and selecting the skip option.
  Unsupervised rules:
  A) Name (10) Last name (10) Treshold (30)
  B) Email (10) Treshold (10)
  
  General rules:
  C) Name (10) Last name (10) Treshold (20)
  D) Name (10) Last name (10) Mail (5) Treshold (25)
  
  Then:
  A + C = duplicate entry. If the general rule would have been applied correctly should not been duplicated, just skipped.
  A + D = duplicate entry.
  B + C = skip the entry (works fine)
  B + D = skip the entry (works fine)
  So as username_2 said, the unsupervised dedupe rule always run, and from there, run your dedupe as well. 
  So the import first check the unsupervised rule and if the Treshold is greater than the weight of the fields always will import the contact independently the general rule you select.
  
  **Tags:** import, duplicate-contacts
  
  > **Comments:**
  > **username_1**: Can you confirm how you are sure which rule is ends up using? Note that if you have a Contact with more than one of the same email, then each instance of that email is counted towards the Threshold. Hence we recommend something like First 10, Last 10, Email 1, Threshold 21 and not F 10, L10, E10, with T of 30, as the latter will count as a match for a contact that has the same email 3 times (it happens!) even if F and L do not match.  
  > **username_0**: I will do more tests to confirm the problem and being carefull with the weight of each field. When I do this tests i'll edit the main posts and show the results.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2018-05-09. Score: 6
  
  The import code's use of dedupe rules as of this writing (Civi 5.1) is...interesting.
  
  The rules, which are defined in `CRM_Contact_Import_Parser_Contact::getPossibleContactMatches` involve short-circuiting the dedupe matching if the external identifier is a match. After that, it will ALSO run the unsupervised dedupe - and from there, run your dedupe as well.
  
  While it'd be great to fix this (it's [happening here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contact/Import/Parser/Contact.php#L2053)) realistically it will take someone putting in resources (coding time or hiring a coder) to resolve this.
  
  For now, the workaround is to modify your unsupervised rule.
  
  > **Comments:**
  > **username_0**: Should I open an issue at jira to report this behavior?  
  > **username_3**: Still true as of Civi 5.12.1. Note the justification for checking the external ID: it's a unique key for contacts, so if the new record matches on external ID, the only two options are to merge/overwrite the new record onto the matching existing record, or to reject the new record altogether. The code chooses to reject the new record if other dedupe criteria would lead to two records having the same external ID.  
  
  ---
  
  ### Answer by username_4 on 2021-01-13. Score: 1
  
  Still having this issue in January 2021, with version 5.32.1 ***But the suggested workaround works fine.***
  
  To summarize the long-winded explanation below: **When trying to import contacts using a dedupe rule of External ID Only, it will not work as expected.** ***An effective workaround is to temporarily replace the unsupervised rule (Email only) with the desired rule (External Id Only).*** Things work as expected, and afterward, unsupervised rule can be changed back to (Email only).
  
  Our problem - an organization has been keeping a flat file in a spreadsheet for years, with columns for various meetings, volunteer dates, etc.
  So trying to import 2400 rows of data with about 50 columns of sparse data. Individual constituents can appear multiple times in the rows, and rows are in order by when the data was collected, making information in a particular row more recent (accurate) than in prior rows. The desired total information import for a contact would then be the latest information in each column, and where the latest row is empty for a given column, then the data in the next most recent row is used, and so on.
  
  So we have made up a unique External ID for each row that effectively uniquely identifies each individual. Since we already have contacts being generated by program registrations through CiviEvent, we want to have the info in the system (e.g. email address - this latest piece of data is presumed to be more accurate than earlier transactions with this person) be kept, but have any blank fields filled in from the most recent row for that contact, (i.e. the "fill" option in contact import), etc. until there are no more rows for that contact.
  
  Our import spreadsheet is in ascending order of External ID, and then secondarily in descending order of the order that was in the original spreadsheet. Thus, for e.g. External ID 24, there might be three rows (e.g. 31-33) of data for that individual, and so the import spreadsheet has ID 24 in three rows, which in the import spreadsheet are in the order of 33, 32, 31. That is, for a given External ID, the most recent data is encountered in the import spreadsheet first, then the second most recent second, etc.
  
  So we set up a dedupe rule "External ID only" and use it with the "fill" option. We presumed that the data in the system (where we had manually added the correct external id) would prevail, augmented by any columns in 33, then 32, then 31, where only the first (latest) information would be retained.
  
  **We were then surprised that the import didn't work, and only a few of the contacts were being imported with a test file of the first 100 rows**. Finding this issue here on Stack Exchange, we realized we were being affected by something similar (or the same - we haven't looked at the code as suggested).
  
  **We used the workaround suggested; changed the unsupervised rule** (email only) to general, created a new rule (External ID only) and made it the unsupervised rule, then did our import. It works exactly as expected. After the big import is completed, we will change the unsupervised rule back to email only (so that people signing up for events won't all result in duplicate contacts, etc.)
  
  
  ---
  
  ### Answer by username_5 on 2018-06-19. Score: 0
  
  Possibly related is that when Skip is selected (the default option) Contact ID is not shown as a match possibility. Contact ID does show when Fill or Update or No Dedupe is selected. This seems like a bug. (When was Fill added to the options? did that cause regression?)
  
  
  ---
  
  