Answers.Body: <p>I think Parvez's work on deduplication performance is a big piece of this puzzle: <a href="http://vedaconsulting.co.uk/civicrm-dedupe-workflow-ui" rel="nofollow noreferrer">http://vedaconsulting.co.uk/civicrm-dedupe-workflow-ui</a> (see step 1).  Some of the answers <a href="https://civicrm.stackexchange.com/questions/3143/batch-merge-automation-of-big-data-set">here</a> are also relevant to improving performance.</p>
  
  <p>However, neither of these solves what's probably your most common use case - someone running a massive deduplication script.  One solution might be to feed deduplication into a queueing system.  Another might be to permission the dedupe screen - or even permission specific dedupe jobs, so that someone can run Parvez's smart dedupe but not a standard dedupe rule.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-06-19T18:41:27.013Z
Answers.Text: Yes, this is a good answer to a complementary question. What I really want is a way to intercept the request, do a calculation on how big a request it is, and stop with an error message if it's too big. I'd guess the right place to do that is the form submit hook. It might be nice to make the maximum number of comparisons configurable. The other thing to do is ensure that the criteria is going to be reasonable - e.g. a non-zero match total (which should probably be in core, anyway).
Answers.UserDisplayName: Alan Dixon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-06-12T20:23:35.667Z
Answers.LastActivityDate: 2015-06-12T20:23:35.667Z
Answers.LastEditDate: 2017-04-13T12:53:13.167Z
Answers.UserDisplayName: Jon G - Megaphone Tech
Answers.Body: <p>It occurs to me that when you can't find the appropriate screwdriver, there is of course the hammer. </p>
  
  <p>Here's one version of a hammer:
  <a href="https://stackoverflow.com/questions/1189039/how-do-i-routinely-kill-mysql-queries-that-have-been-alive-for-too-long">https://stackoverflow.com/questions/1189039/how-do-i-routinely-kill-mysql-queries-that-have-been-alive-for-too-long</a></p>
  
  <p>It'd be nicer to have an extension that analyses the sql before executing it and catching/analysing it.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-04-25T16:13:56.670Z
Answers.LastActivityDate: 2016-09-06T20:48:34.230Z
Answers.LastEditDate: 2017-05-23T12:38:59.763Z
Answers.UserDisplayName: Alan Dixon
Body: <p>On a reasonably big data set (e.g. one with 70,000 individual contacts), it's not hard to kill your server with a naive manual de-duplication attempt. For example, if you don't specify a group, that number of contacts generates about 5 billion comparisons.</p>
  
  <p>Has anyone thought of a way to prevent unreasonable deduplication attempts?</p>
  
Tags: <mysql><duplicate-contacts>
Title: How can I prevent deduping from killing my server?
ThreadText: # How can I prevent deduping from killing my server?
  
  (Asked by: username_0 on 2015-06-12)
  
  On a reasonably big data set (e.g. one with 70,000 individual contacts), it's not hard to kill your server with a naive manual de-duplication attempt. For example, if you don't specify a group, that number of contacts generates about 5 billion comparisons.
  
  Has anyone thought of a way to prevent unreasonable deduplication attempts?
  
  **Tags:** mysql, duplicate-contacts
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2015-06-12. Score: 2
  
  I think Parvez's work on deduplication performance is a big piece of this puzzle: <http://vedaconsulting.co.uk/civicrm-dedupe-workflow-ui> (see step 1). Some of the answers [here](https://civicrm.stackexchange.com/questions/3143/batch-merge-automation-of-big-data-set) are also relevant to improving performance.
  
  However, neither of these solves what's probably your most common use case - someone running a massive deduplication script. One solution might be to feed deduplication into a queueing system. Another might be to permission the dedupe screen - or even permission specific dedupe jobs, so that someone can run Parvez's smart dedupe but not a standard dedupe rule.
  
  > **Comments:**
  > **username_0**: Yes, this is a good answer to a complementary question. What I really want is a way to intercept the request, do a calculation on how big a request it is, and stop with an error message if it's too big. I'd guess the right place to do that is the form submit hook. It might be nice to make the maximum number of comparisons configurable. The other thing to do is ensure that the criteria is going to be reasonable - e.g. a non-zero match total (which should probably be in core, anyway).  
  
  ---
  
  ### Answer by username_0 on 2016-04-25. Score: 0
  
  It occurs to me that when you can't find the appropriate screwdriver, there is of course the hammer.
  
  Here's one version of a hammer:
  <https://stackoverflow.com/questions/1189039/how-do-i-routinely-kill-mysql-queries-that-have-been-alive-for-too-long>
  
  It'd be nicer to have an extension that analyses the sql before executing it and catching/analysing it.
  
  
  ---
  
  