Answers.Body: <p>Per Karin's advice, filed bug report on Core - <a href="https://lab.civicrm.org/dev/core/-/issues/1767" rel="nofollow noreferrer">https://lab.civicrm.org/dev/core/-/issues/1767</a>. Filed a separate one for webform_civicrm - <a href="https://www.drupal.org/project/webform_civicrm/issues/3137284" rel="nofollow noreferrer">https://www.drupal.org/project/webform_civicrm/issues/3137284</a>. Both are needed for our use case.
  I proposed patches with each issue, should anyone else run into this before it is fixed properly. BTW, good to know that the phone match is based on digits only. All formatting &amp; decorations are stripped off.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T21:04:44.027Z
Answers.Text: Nice - well done - 
Answers.UserDisplayName: KarinG - Semper IT
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T21:50:44.120Z
Answers.Text: Sorry @KarinG-SemperIT, had to patch webform_civicrm as well 
Answers.UserDisplayName: wil_SRQ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T23:49:51.013Z
Answers.Text: No problem - happy you managed to work through it! We'll do some thinking to see which way we should go on this.
Answers.UserDisplayName: KarinG - Semper IT
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T16:48:26.537Z
Answers.LastActivityDate: 2020-05-16T21:48:39.900Z
Answers.LastEditDate: 2020-05-16T21:48:39.900Z
Answers.UserDisplayName: wil_SRQ
Body: <p>The snippets below show the duplicate contacts, the webform CiviCRM tab, and the matching rules. If I fill in name &amp; a matching email the form updates the contact, does not create a dup. If I supply name &amp; a matching phone the duplicate shows up. The Civi and Drupal logs don't show anything related. Thoughts?</p>
  
  <p>Civi v5.24.3, Drupal v7.69, PHP v7.1.14, MySQL v5.6.41-84.1, Webform v7.x-4.22, Webform CiviCRM v7.x-4.28.</p>
  
  <p><a href="https://i.sstatic.net/hZl2C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hZl2C.png" alt="Duplicate contacts"></a></p>
  
  <p><a href="https://i.sstatic.net/3FdKX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3FdKX.png" alt="Webform CiviCRM tab"></a></p>
  
  <p><a href="https://i.sstatic.net/fWcR8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fWcR8.png" alt="Matching rules"></a></p>
  
  <p><a href="https://i.sstatic.net/JZZ6C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JZZ6C.png" alt="Default Unsupervised matching rule for Individual"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T13:08:42.247Z
Comments.Text: i) Is this dedupe rule working on your front end - CiviCRM native pages? Webform CiviCRM just uses CiviCRM APIs. ii) upgrade to 7.x-5.x - we have replaced some leftover direct queries with APIs, not sure if dedupe was one of them iii) I've never used Phone # in a dedupe rule as users can enter their digits in so many ways, with and without (), -, +1, so many options. I know you're testing and matching it here - but even if you get it working - it will be tough to make it useful. iii) then there are all the type and location id-s on phone numbers. This makes for a really complicated dedupe rule
Comments.UserDisplayName: KarinG - Semper IT
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T13:10:25.383Z
Comments.Text: PS - upvote for the great screenshots!
Comments.UserDisplayName: KarinG - Semper IT
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T18:01:56.053Z
Comments.Text: Glad the screenshots help! Updated webform_civicrm to 7.x-5.0 with the same duplicate results. If you meant 7.x-5.x-dev let me know and I'll try that too. If I "Use Rule" in CiviCRM's Find and Merge Duplicate Contacts this pair of duplicates is called out as expected.
  I was curious whether the match would be based on the string or on something deeper, i.e. civicrm_phone.phone_numeric in the dB. In any case, my numbers are all in Florida and I can force the input to 10 digits. Many of the parents we work with don't use email so we're transitioning to rely on mobile phones instead.
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T18:55:55.527Z
Comments.Text: No - not -dev - just 7.x-5.x brach is ok. There have been recent fixes to the dedupe API in CiviCRM core that related to numbers (dob) and also custom fields. Have you tried creating a profile with these fields and fill it out (while logged out) so you can test your unsupervised dedupe rule?
Comments.UserDisplayName: KarinG - Semper IT
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T22:49:44.230Z
Comments.Text: No, I've never used profiles. But this got much scarier now. I have another CiviCRM site and a test form there performs perfectly. So something is broken in the first site and I have no idea where to start looking :-(
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T23:04:39.777Z
Comments.Text: Ok, I created a profile and it invariably reates a new contact. Hopefully I did it right, in "What to do upon duplicate match " selected "Update the matching contact"
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T23:32:26.060Z
Comments.Text: Then I created a profile in the 2nd system and it invariably updates the existing contact. So the 2nd system behaves as hoped with both form and profile, while the 1st system creates dupes with both. I can't find any differences in the matching rules, but continue looking.
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T01:52:53.607Z
Comments.Text: Differences in CiviCRM versions; CiviCRM Extensions, custom php code? Good luck hunting that down... that can be tricky.
Comments.UserDisplayName: KarinG - Semper IT
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T05:22:25.757Z
Comments.Text: Just a comment on your matching rule. if a record happens to have 3 times the same email then it would record as a MATCH even if the First and Last were different. Not suggesting this is your issue, but a suggestion that a rule set as eg 10+10+2 will not fail in such circumstances where a rule with 10+10+10 will.
Comments.UserDisplayName: petednz - fuzion
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T10:16:37.337Z
Comments.Text: @petednz-fuzion - did you come up with that gotcha by yourself? Or did you experience it? Impressively obscure!
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T10:17:54.113Z
Comments.Text: @KarinG-SemperIT - I found the differences and, unfortunately, the error was w. the 2nd system. It wasn't using Phone as a match field correctly. My bad, should've spotted it earlier.
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T10:24:35.820Z
Comments.Text: Since both my systems are now failing to match on phone, I tested in a 3rd system - https://corona.civicrm.jaapjansma.nl/ admin:admin1234 courtesy of @Jaap Jansma - CiviCooP. It also failed to match on phone :-( Created base contact (dupe_left), profile (dupe test) and rule (Name Email Phone). Temporarily made the new rule the Unsupervised default and the profile created a duplicate if email was left blank.
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T12:27:15.930Z
Comments.Text: So to summarize I think you have found a bug in dedupe rules using (phone) numbers in CiviCRM Core itself -> time to file a Gitlab ticket/issue -> bonus points for reproducing this issue [and for this purpose make the unsupervised rule -> phone number only] on https://dmaster.demo.civicrm.org/ [and post screenshots from that -> in your Gitlab ticket].
Comments.UserDisplayName: KarinG - Semper IT
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T16:56:22.370Z
Comments.Text: @KarinG-SemperIT - answered with GitLab issue, thanks for your guidance throughout. I REALLY tried to get helpful screenshots from dmaster.demo.civicrm.org but couldn't get it to display Phone in basic nor advanced search results. The field is there but grayed out and blank! I even tried a Contact Detail Report, same result. There's something funny going on with some pieces of the interface being in French (e.g. try adding first name to a profile). May or may not relate to the Phone display.
Comments.UserDisplayName: wil_SRQ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-17T06:03:00.140Z
Comments.Text: we spotted some clients dedupes were not firing and this was the issue - yes obscure but a lot of Rules get built such that is has this potential to misfire
Comments.UserDisplayName: petednz - fuzion
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-17T06:03:16.367Z
Comments.Text: i just tried dmaster and phone is searchable again via Adv. Search
Comments.UserDisplayName: petednz - fuzion
Tags: <webform-civicrm><dedupe-rules>
Title: Webform creating duplicate contact
ThreadText: # Webform creating duplicate contact
  
  (Asked by: username_0 on 2020-05-14)
  
  The snippets below show the duplicate contacts, the webform CiviCRM tab, and the matching rules. If I fill in name & a matching email the form updates the contact, does not create a dup. If I supply name & a matching phone the duplicate shows up. The Civi and Drupal logs don't show anything related. Thoughts?
  
  Civi v5.24.3, Drupal v7.69, PHP v7.1.14, MySQL v5.6.41-84.1, Webform v7.x-4.22, Webform CiviCRM v7.x-4.28.
  
  [![Duplicate contacts](https://i.sstatic.net/hZl2C.png)](https://i.sstatic.net/hZl2C.png)
  
  [![Webform CiviCRM tab](https://i.sstatic.net/3FdKX.png)](https://i.sstatic.net/3FdKX.png)
  
  [![Matching rules](https://i.sstatic.net/fWcR8.png)](https://i.sstatic.net/fWcR8.png)
  
  [![Default Unsupervised matching rule for Individual](https://i.sstatic.net/JZZ6C.png)](https://i.sstatic.net/JZZ6C.png)
  
  **Tags:** webform-civicrm, dedupe-rules
  
  > **Comments:**
  > **username_1**: i) Is this dedupe rule working on your front end - CiviCRM native pages? Webform CiviCRM just uses CiviCRM APIs. ii) upgrade to 7.x-5.x - we have replaced some leftover direct queries with APIs, not sure if dedupe was one of them iii) I've never used Phone # in a dedupe rule as users can enter their digits in so many ways, with and without (), -, +1, so many options. I know you're testing and matching it here - but even if you get it working - it will be tough to make it useful. iii) then there are all the type and location id-s on phone numbers. This makes for a really complicated dedupe rule  
  > **username_1**: PS - upvote for the great screenshots!  
  > **username_0**: Glad the screenshots help! Updated webform_civicrm to 7.x-5.0 with the same duplicate results. If you meant 7.x-5.x-dev let me know and I'll try that too. If I "Use Rule" in CiviCRM's Find and Merge Duplicate Contacts this pair of duplicates is called out as expected. I was curious whether the match would be based on the string or on something deeper, i.e. civicrm_phone.phone_numeric in the dB. In any case, my numbers are all in Florida and I can force the input to 10 digits. Many of the parents we work with don't use email so we're transitioning to rely on mobile phones instead.  
  > **username_1**: No - not -dev - just 7.x-5.x brach is ok. There have been recent fixes to the dedupe API in CiviCRM core that related to numbers (dob) and also custom fields. Have you tried creating a profile with these fields and fill it out (while logged out) so you can test your unsupervised dedupe rule?  
  > **username_0**: No, I've never used profiles. But this got much scarier now. I have another CiviCRM site and a test form there performs perfectly. So something is broken in the first site and I have no idea where to start looking :-(  
  > **username_0**: Ok, I created a profile and it invariably reates a new contact. Hopefully I did it right, in "What to do upon duplicate match " selected "Update the matching contact"  
  > **username_0**: Then I created a profile in the 2nd system and it invariably updates the existing contact. So the 2nd system behaves as hoped with both form and profile, while the 1st system creates dupes with both. I can't find any differences in the matching rules, but continue looking.  
  > **username_1**: Differences in CiviCRM versions; CiviCRM Extensions, custom php code? Good luck hunting that down... that can be tricky.  
  > **username_2**: Just a comment on your matching rule. if a record happens to have 3 times the same email then it would record as a MATCH even if the First and Last were different. Not suggesting this is your issue, but a suggestion that a rule set as eg 10+10+2 will not fail in such circumstances where a rule with 10+10+10 will.  
  > **username_0**: @petednz-fuzion - did you come up with that gotcha by yourself? Or did you experience it? Impressively obscure!  
  > **username_0**: @KarinG-SemperIT - I found the differences and, unfortunately, the error was w. the 2nd system. It wasn't using Phone as a match field correctly. My bad, should've spotted it earlier.  
  > **username_0**: Since both my systems are now failing to match on phone, I tested in a 3rd system - https://corona.civicrm.jaapjansma.nl/ admin:admin1234 courtesy of @Jaap Jansma - CiviCooP. It also failed to match on phone :-( Created base contact (dupe_left), profile (dupe test) and rule (Name Email Phone). Temporarily made the new rule the Unsupervised default and the profile created a duplicate if email was left blank.  
  > **username_1**: So to summarize I think you have found a bug in dedupe rules using (phone) numbers in CiviCRM Core itself -> time to file a Gitlab ticket/issue -> bonus points for reproducing this issue [and for this purpose make the unsupervised rule -> phone number only] on https://dmaster.demo.civicrm.org/ [and post screenshots from that -> in your Gitlab ticket].  
  > **username_0**: @KarinG-SemperIT - answered with GitLab issue, thanks for your guidance throughout. I REALLY tried to get helpful screenshots from dmaster.demo.civicrm.org but couldn't get it to display Phone in basic nor advanced search results. The field is there but grayed out and blank! I even tried a Contact Detail Report, same result. There's something funny going on with some pieces of the interface being in French (e.g. try adding first name to a profile). May or may not relate to the Phone display.  
  > **username_2**: we spotted some clients dedupes were not firing and this was the issue - yes obscure but a lot of Rules get built such that is has this potential to misfire  
  > **username_2**: i just tried dmaster and phone is searchable again via Adv. Search  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-16. Score: 3
  
  Per Karin's advice, filed bug report on Core - <https://lab.civicrm.org/dev/core/-/issues/1767>. Filed a separate one for webform\_civicrm - <https://www.drupal.org/project/webform_civicrm/issues/3137284>. Both are needed for our use case.
  I proposed patches with each issue, should anyone else run into this before it is fixed properly. BTW, good to know that the phone match is based on digits only. All formatting & decorations are stripped off.
  
  > **Comments:**
  > **username_1**: Nice - well done -   
  > **username_0**: Sorry @KarinG-SemperIT, had to patch webform_civicrm as well   
  > **username_1**: No problem - happy you managed to work through it! We'll do some thinking to see which way we should go on this.  
  
  ---
  
  