Answers.Body: <p>Do you have the batch merge dedupe scheduled job enabled?</p>
  <p><a href="https://dmaster.demo.civicrm.org/civicrm/admin/job/edit?action=update&amp;id=21&amp;reset=1" rel="nofollow noreferrer">https://dmaster.demo.civicrm.org/civicrm/admin/job/edit?action=update&amp;id=21&amp;reset=1</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T15:46:18.313Z
Answers.Text: Nope, the job is not even in the list of enabled jobs - neither in the jobs list at all .
Answers.UserDisplayName: yurg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T16:02:37.540Z
Answers.Text: That's strange. It's part of core and not  new at all. Try Ctrl-F and search for process_batch_merge on the jobs page?
Answers.UserDisplayName: Demerit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T17:14:16.427Z
Answers.Text: Try `cv api Job.process_batch_merge gid=foo`. If it says gid is not a valid integer, then it means it recognizes the job exists, and something somewhere is probably calling it. You could grep the filesystem for process_batch_merge. If it says the job does not exist then that would be strange, but would mean something else is going on.
Answers.UserDisplayName: Demerit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T18:52:00.580Z
Answers.Text: Yes, it's definitely "something else": **cv api Job.get**  output has missing jobs between id17 and id22, However I don't understand, if a job doesn't exist, how would it be executed.
Answers.UserDisplayName: yurg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T19:26:32.083Z
Answers.Text: There's a difference between a job action and an job instance - a little bit like civireport. Job.get returns the saved instances. But you can still run the action directly like the example call I gave.
Answers.UserDisplayName: Demerit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T20:19:14.787Z
Answers.Text: I see. The demo link https://dmaster.demo.civicrm.org/civicrm/admin/job/edit?action=update&id=21&reset=1 states the job ID is 21 therefore I thought the "foo" has to be job id, eg. 21 And indeed, there is the  _"error_message": "gid is not a valid integer"._ after the `cv api Job.process_batch_merge gid=foo`
Answers.UserDisplayName: yurg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T20:25:04.970Z
Answers.Text: `grep -irl "process_batch_merge"` returns  wp-content/
  wp-content/plugins/civicrm/civicrm/api/v3/Job.php
  wp-content/plugins/civicrm/civicrm/CRM/Core/Permission.php
  wp-content/plugins/civicrm/civicrm/sql/civicrm_generated.mysql
  wp-content/plugins/civicrm/civicrm/sql/civicrm_data.mysql
  wp-content/plugins/civicrm/civicrm/sql/civicrm_data/civicrm_job.sqldata.php Does it mean the **civicrm/api/v3/Job.php** gets executed even if the job is not in the database?
Answers.UserDisplayName: yurg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T21:19:29.067Z
Answers.Text: No it would only be if something is calling it directly. Maybe you have a separate cron setup? Check crontab or if you set up cron via a web UI then check in there.
Answers.UserDisplayName: Demerit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-03T13:48:49.733Z
Answers.Text: Afraid it's something else: brand new migration from D7 towith a single record executing all scheduled jobs via **cv**
Answers.UserDisplayName: yurg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-03T15:08:58.970Z
Answers.Text: Ok try adding a line `CRM_Core_Error::backtrace('hi', true);`  just before this line to give a backtrace: https://github.com/civicrm/civicrm-core/blob/cf173170734ea36f325b14410b30e6d15ddb31c1/CRM/Dedupe/BAO/DedupeRuleGroup.php#L238. Then check the civi log after cron runs. This should tell what is running it.
Answers.UserDisplayName: Demerit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-04T11:24:01.880Z
Answers.Text: There are  new things in logs: "Cannot determine api action for CRM_Group_Form_Edit.CRM_Core_Action "NO DESCRIPTION SET" not recognized."" but this looks not relevant to the case.
Answers.UserDisplayName: yurg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T15:36:47.680Z
Answers.LastActivityDate: 2024-10-02T15:36:47.680Z
Answers.UserDisplayName: Demerit
Body: <p>Something kills/freezes MySQL once per cron by executing:</p>
  <blockquote>
  <p>INSERT INTO civicrm_tmp_e_dedupe_301d41956c9e2bd93bf184f95b40adef  (id1, id2, weight) SELECT id1, id2, weight FROM (SELECT t1.contact_id id1, t2.contact_id id2, 20 weight FROM civicrm_email t1 INNER JOIN civicrm_email t2 ON ((t1.email IS NOT NULL AND t2.email IS NOT NULL AND t1.email = t2.email AND t1.email &lt;&gt; '' AND t2.email &lt;&gt; '')) WHERE t1.contact_id &lt; t2.contact_id AND t1.contact_id IN (3,5,9,11,19,21,25,28,38....<strong>SOME 100K IDs here</strong> .....102486,102487,102488,102489,102490,102498,102499)) subunion GROUP BY id1, id2, weight ON DUPLICATE KEY UPDATE weight = weight + VALUES(weight)</p>
  </blockquote>
  <p>There are only about 70K contacts in the database, nothing related in Scheduled Jobs, nor any custom Wordpress Plugin.
  CiviCRM Rules not installed nor any extensions which would look like possible interceptors.  There is only one crontab entry:</p>
  <blockquote>
  <p>/usr/local/bin/php  /home/username/cv api job.execute --user=civicrm</p>
  </blockquote>
  <p>It feels like some &quot;duplicate finder&quot; script has been launched from somewhere and forgotten.</p>
  <p>There is something possible related: a few (not all, just 3 or 4 out of a dozen total) Wordpress admins, having CiviCRM administrator role, got thousands duplicated contact records in CiviCRM.</p>
  <p>Ran out of ideas currently apart from the database search - wish I knew what to search for though.</p>
  <p><em>Wordpress the latest,  CiviCRM 5.76.2,</em></p>
  <p><a href="https://i.sstatic.net/IYqp9xhW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IYqp9xhW.png" alt="Sheduled jobs" /></a></p>
  
Tags: <duplicate-contacts><dedupe-rules>
Title: Where does the "INSERT INTO civicrm_tmp_e_dedupe_" come from?
ThreadText: # Where does the "INSERT INTO civicrm_tmp_e_dedupe_" come from?
  
  (Asked by: username_0 on 2024-10-02)
  
  Something kills/freezes MySQL once per cron by executing:
  
  > INSERT INTO civicrm\_tmp\_e\_dedupe\_301d41956c9e2bd93bf184f95b40adef (id1, id2, weight) SELECT id1, id2, weight FROM (SELECT t1.contact\_id id1, t2.contact\_id id2, 20 weight FROM civicrm\_email t1 INNER JOIN civicrm\_email t2 ON ((t1.email IS NOT NULL AND t2.email IS NOT NULL AND t1.email = t2.email AND t1.email <> '' AND t2.email <> '')) WHERE t1.contact\_id < t2.contact\_id AND t1.contact\_id IN (3,5,9,11,19,21,25,28,38....**SOME 100K IDs here** .....102486,102487,102488,102489,102490,102498,102499)) subunion GROUP BY id1, id2, weight ON DUPLICATE KEY UPDATE weight = weight + VALUES(weight)
  
  There are only about 70K contacts in the database, nothing related in Scheduled Jobs, nor any custom Wordpress Plugin.
  CiviCRM Rules not installed nor any extensions which would look like possible interceptors. There is only one crontab entry:
  
  > /usr/local/bin/php /home/username/cv api job.execute --user=civicrm
  
  It feels like some "duplicate finder" script has been launched from somewhere and forgotten.
  
  There is something possible related: a few (not all, just 3 or 4 out of a dozen total) Wordpress admins, having CiviCRM administrator role, got thousands duplicated contact records in CiviCRM.
  
  Ran out of ideas currently apart from the database search - wish I knew what to search for though.
  
  *Wordpress the latest, CiviCRM 5.76.2,*
  
  [![Sheduled jobs](https://i.sstatic.net/IYqp9xhW.png)](https://i.sstatic.net/IYqp9xhW.png)
  
  **Tags:** duplicate-contacts, dedupe-rules
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-02. Score: 0
  
  Do you have the batch merge dedupe scheduled job enabled?
  
  <https://dmaster.demo.civicrm.org/civicrm/admin/job/edit?action=update&id=21&reset=1>
  
  > **Comments:**
  > **username_0**: Nope, the job is not even in the list of enabled jobs - neither in the jobs list at all .  
  > **username_1**: That's strange. It's part of core and not  new at all. Try Ctrl-F and search for process_batch_merge on the jobs page?  
  > **username_1**: Try `cv api Job.process_batch_merge gid=foo`. If it says gid is not a valid integer, then it means it recognizes the job exists, and something somewhere is probably calling it. You could grep the filesystem for process_batch_merge. If it says the job does not exist then that would be strange, but would mean something else is going on.  
  > **username_0**: Yes, it's definitely "something else": **cv api Job.get**  output has missing jobs between id17 and id22, However I don't understand, if a job doesn't exist, how would it be executed.  
  > **username_1**: There's a difference between a job action and an job instance - a little bit like civireport. Job.get returns the saved instances. But you can still run the action directly like the example call I gave.  
  > **username_0**: I see. The demo link https://dmaster.demo.civicrm.org/civicrm/admin/job/edit?action=update&id=21&reset=1 states the job ID is 21 therefore I thought the "foo" has to be job id, eg. 21 And indeed, there is the  _"error_message": "gid is not a valid integer"._ after the `cv api Job.process_batch_merge gid=foo`  
  > **username_0**: `grep -irl "process_batch_merge"` returns  wp-content/ wp-content/plugins/civicrm/civicrm/api/v3/Job.php wp-content/plugins/civicrm/civicrm/CRM/Core/Permission.php wp-content/plugins/civicrm/civicrm/sql/civicrm_generated.mysql wp-content/plugins/civicrm/civicrm/sql/civicrm_data.mysql wp-content/plugins/civicrm/civicrm/sql/civicrm_data/civicrm_job.sqldata.php Does it mean the **civicrm/api/v3/Job.php** gets executed even if the job is not in the database?  
  > **username_1**: No it would only be if something is calling it directly. Maybe you have a separate cron setup? Check crontab or if you set up cron via a web UI then check in there.  
  > **username_0**: Afraid it's something else: brand new migration from D7 towith a single record executing all scheduled jobs via **cv**  
  > **username_1**: Ok try adding a line `CRM_Core_Error::backtrace('hi', true);`  just before this line to give a backtrace: https://github.com/civicrm/civicrm-core/blob/cf173170734ea36f325b14410b30e6d15ddb31c1/CRM/Dedupe/BAO/DedupeRuleGroup.php#L238. Then check the civi log after cron runs. This should tell what is running it.  
  > **username_0**: There are  new things in logs: "Cannot determine api action for CRM_Group_Form_Edit.CRM_Core_Action "NO DESCRIPTION SET" not recognized."" but this looks not relevant to the case.  
  
  ---
  
  