Answers.Body: <h2>Some thoughts</h2>
  <p>Anonymizing data is hard work.</p>
  <p>Eg: Would it be OK if your developer could de-anonymize your contacts by identifying from geolocated data? What about identifying by &quot;metadata&quot;, if suburb data is left in? Does your developer need a DB with activities from inbound email left in? Has the situation improved in the decade since <a href="http://archive.wired.com/politics/security/commentary/securitymatters/2007/12/securitymatters_1213" rel="nofollow noreferrer"><code>WIRED</code> Why 'Anonymous' data sometimes isn't</a>, or gotten worse? (It got worse!)</p>
  <p>As you anticipated, generating real-looking fake data is much slower than garbage data, and that's one tradeoff. The other is that a CRM has data in places you might not think to look, so you are safer treating the data as &quot;mostly anonymized&quot; even when you're done. (See that Wired article!)</p>
  <p>One approach, which may also speed up your testing, is to remove data unless you know it's needed (and only anonymize what you didn't already remove).</p>
  <ul>
  <li>Not testing mailing reports? <em>Truncate those huge mailing tables.</em></li>
  <li>Not testing custom data? <em>Great, that stuff is hard to anonymize! Drop it!</em></li>
  <li>Not testing performance? <em>Cull your test DB down to 500 contacts before anonymizing.</em></li>
  </ul>
  <p>What you choose depends on how you need to use it, and what's OK for your testing. As you say, for debugging developers need a &quot;near exact&quot; copy of the live data ... but the more a test data set matches your site's profile, the less anonymous it will be.</p>
  <h2>Some starting points</h2>
  <ul>
  <li><strong>anonymize extension</strong> (<a href="https://civicrm.org/extensions/anonymize" rel="nofollow noreferrer">anonymize</a>) aims to anonymize a DB. Offers two strategies: Faker (slow, generates real-looking names like &quot;King, Lesley&quot;) and SQL (fast, generates names like &quot;Tdfv, Fnedftd&quot;). With recent updates, this is probably your best bet - but you'll be an early user, so <a href="https://lab.civicrm.org/extensions/anonymize/-/issues" rel="nofollow noreferrer">please direct feedback to the issue queue in CiviCRM's Gitlab</a>!</li>
  <li><strong>gibberish extension</strong> (<a href="https://civicrm.org/extensions/gibberish" rel="nofollow noreferrer">gibberish</a>) creates SQL to replace the contents of your database with gibberish</li>
  <li><strong>Michael McAndrew's <a href="https://github.com/michaelmcandrew/pop" rel="nofollow noreferrer">POP library</a></strong> is the most featureful IMO, its goal seems to be to populate a site with fake data rather than anonymize an existing DB. See the blog post <a href="https://civicrm.org/blog/michael-mcandrew/populate-your-civicrm-site-with-fake-data" rel="nofollow noreferrer">Populate your CiviCRM site with fake data</a>. A similar example using nodejs is <a href="https://github.com/tttp/civi-charlatan" rel="nofollow noreferrer">civi-charlatan</a>.</li>
  <li><strong>anonymiser</strong> (<a href="https://github.com/systopia/de.systopia.anonymiser" rel="nofollow noreferrer">systopia/de.systopia.anonymiser</a>) is another CiviCRM anonymiser extension under recent development as of June 2018. I haven't checked this out (and Sean and I didn't find it when we started work on anonymizer).</li>
  <li><strong>redaction_tool</strong> (<a href="https://github.com/JohnFF/redaction_tool" rel="nofollow noreferrer">johnff/redaction_tool</a>) is another tool to redact contact data (WIP)</li>
  <li><strong>civicrm-fake-data</strong> (on <a href="https://github.com/systopia/civicrm-fake-data" rel="nofollow noreferrer">github</a>) is a ruby script to replace existing data with fake data.</li>
  <li><strong>civicrmanonymisation</strong> is an <a href="https://github.com/compucorp/civicrmanonymisation/" rel="nofollow noreferrer">SQL approach to anonymising a DB</a>.</li>
  <li><strong>encryptDB.php</strong> <em>was</em> <a href="https://github.com/civicrm/civicrm-core/commit/da08fd08011964b6ad2187061a89f720af20106a" rel="nofollow noreferrer">in CiviCRM core</a>, but it was a poor starting point; MD5 is not encryption, so what changes it does make are reversible. It also targeted a limited number of fields, so your DB would still contain identifying data even if not deanonymised ...</li>
  <li><strong>civicrm_obfuscate_db.php</strong> (based on <code>encryptDB.php</code> above) is another script I wrote several years back. It generates a <a href="https://github.com/xurizaemon/org.civicrm.contrib.anonymize/blob/obfuscate_db/obfuscate_db.php" rel="nofollow noreferrer">set of SQL commands to obfuscate your DB</a>. It's SQL-based so relatively fast, generates incomprehensible names (&quot;bggerns srrssbb&quot;), and like all the above options is not a complete solution.</li>
  <li>Another option would be to construct a test environment with test only data that simulates all situations you need to test, and keep expanding it as you identify new bugs. Let's be realistic, few CiviCRM sites have budget for that </li>
  </ul>
  <p>To recap: none of the above are complete working solutions for what you want, and the solution you need depends on your context. But please do contribute your experiences to the community - most of the above have a Github issue queue you could submit improvements / feedback to!</p>
  <p>One particular sticking point for several of the above seems to be CiviCRM's &quot;Database Logging&quot; tables, which by default use a format which complicates sanitisation. If you're testing or using any of the above, you should definitely review whether those tables are being de-identified completely in the process.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-28T04:04:31.863Z
Answers.Text: <stands up and starts clapping>
Answers.UserDisplayName: Allen Hutchison
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-28T05:15:56.770Z
Answers.Text: CRM-20346 just got closed.
Answers.UserDisplayName: petednz - fuzion
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-28T10:35:11.237Z
Answers.Text: Chris Burgess, so much effort went into this answer. Really thankful for that! I will surely study the suggested options and give feedback to, in a way, "pay back" your effort.
Answers.UserDisplayName: ñull
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-28T11:18:05.010Z
Answers.Text: Thanks! You aren't "nully", previously of the FSF are you? Guessing not, but if so, I'd love to pick your brains on something.
Answers.UserDisplayName: Chris Burgess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-25T14:02:06.067Z
Answers.Text: A word of warning - make sure that outgoing email is disabled!  You don't want any of your CiviRules etc. being triggered and sending out emails.  I saw a colleague who had [CiviCRM WordPress Profile Sync](https://wordpress.org/plugins/civicrm-wp-profile-sync/) installed, which meant that changing the email address in Civi changed the email in WordPress - which sent out a "your email address has been changed" message to everyone who had a login...
Answers.UserDisplayName: Jon G - Megaphone Tech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-03-28T01:30:18.420Z
Answers.LastActivityDate: 2022-08-27T19:23:01.133Z
Answers.LastEditDate: 2022-08-27T19:23:01.133Z
Answers.UserDisplayName: Chris Burgess
Body: <p>Private information is confidential. However for developers to debug errors they need a copy of your data that's close enough to production to effectively debug. How can we redact, strip or remove confidential data from CiviCRM?</p>
  
  <p>Two options come to mind. I could spend time creating anonymous test data that covers  all the possible variants of our complex hierarchical contact relationship structure. The other option would be to go through a copy of production and anonymise it by overwriting private data with random strings. But we are talking here about some 9000 contacts. Working with API calls to anonymise could take hours, not really an option for a fast workflow. </p>
  
  <p>Anyone with experience or ideas could share his thoughts on this subject? </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-03-27T19:26:08.087Z
Comments.Text: I will poke my colleague who did some work on anonymising solutions
Comments.UserDisplayName: petednz - fuzion
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-03-28T04:55:18.437Z
Comments.Text: see what happens when he gets poked!
Comments.UserDisplayName: petednz - fuzion
Tags: <drupal><developer>
Title: How would you anonymise (or redact) private data for developer stage?
ThreadText: # How would you anonymise (or redact) private data for developer stage?
  
  (Asked by: username_0 on 2017-03-27)
  
  Private information is confidential. However for developers to debug errors they need a copy of your data that's close enough to production to effectively debug. How can we redact, strip or remove confidential data from CiviCRM?
  
  Two options come to mind. I could spend time creating anonymous test data that covers all the possible variants of our complex hierarchical contact relationship structure. The other option would be to go through a copy of production and anonymise it by overwriting private data with random strings. But we are talking here about some 9000 contacts. Working with API calls to anonymise could take hours, not really an option for a fast workflow.
  
  Anyone with experience or ideas could share his thoughts on this subject?
  
  **Tags:** drupal, developer
  
  > **Comments:**
  > **username_1**: I will poke my colleague who did some work on anonymising solutions  
  > **username_1**: see what happens when he gets poked!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-03-28. Score: 22
  
  ## Some thoughts
  
  Anonymizing data is hard work.
  
  Eg: Would it be OK if your developer could de-anonymize your contacts by identifying from geolocated data? What about identifying by "metadata", if suburb data is left in? Does your developer need a DB with activities from inbound email left in? Has the situation improved in the decade since [`WIRED` Why 'Anonymous' data sometimes isn't](http://archive.wired.com/politics/security/commentary/securitymatters/2007/12/securitymatters_1213), or gotten worse? (It got worse!)
  
  As you anticipated, generating real-looking fake data is much slower than garbage data, and that's one tradeoff. The other is that a CRM has data in places you might not think to look, so you are safer treating the data as "mostly anonymized" even when you're done. (See that Wired article!)
  
  One approach, which may also speed up your testing, is to remove data unless you know it's needed (and only anonymize what you didn't already remove).
  
  * Not testing mailing reports? *Truncate those huge mailing tables.*
  * Not testing custom data? *Great, that stuff is hard to anonymize! Drop it!*
  * Not testing performance? *Cull your test DB down to 500 contacts before anonymizing.*
  
  What you choose depends on how you need to use it, and what's OK for your testing. As you say, for debugging developers need a "near exact" copy of the live data ... but the more a test data set matches your site's profile, the less anonymous it will be.
  
  ## Some starting points
  
  * **anonymize extension** ([anonymize](https://civicrm.org/extensions/anonymize)) aims to anonymize a DB. Offers two strategies: Faker (slow, generates real-looking names like "King, Lesley") and SQL (fast, generates names like "Tdfv, Fnedftd"). With recent updates, this is probably your best bet - but you'll be an early user, so [please direct feedback to the issue queue in CiviCRM's Gitlab](https://lab.civicrm.org/extensions/anonymize/-/issues)!
  * **gibberish extension** ([gibberish](https://civicrm.org/extensions/gibberish)) creates SQL to replace the contents of your database with gibberish
  * **Michael McAndrew's [POP library](https://github.com/michaelmcandrew/pop)** is the most featureful IMO, its goal seems to be to populate a site with fake data rather than anonymize an existing DB. See the blog post [Populate your CiviCRM site with fake data](https://civicrm.org/blog/michael-mcandrew/populate-your-civicrm-site-with-fake-data). A similar example using nodejs is [civi-charlatan](https://github.com/tttp/civi-charlatan).
  * **anonymiser** ([systopia/de.systopia.anonymiser](https://github.com/systopia/de.systopia.anonymiser)) is another CiviCRM anonymiser extension under recent development as of June 2018. I haven't checked this out (and Sean and I didn't find it when we started work on anonymizer).
  * **redaction\_tool** ([johnff/redaction\_tool](https://github.com/JohnFF/redaction_tool)) is another tool to redact contact data (WIP)
  * **civicrm-fake-data** (on [github](https://github.com/systopia/civicrm-fake-data)) is a ruby script to replace existing data with fake data.
  * **civicrmanonymisation** is an [SQL approach to anonymising a DB](https://github.com/compucorp/civicrmanonymisation/).
  * **encryptDB.php** *was* [in CiviCRM core](https://github.com/civicrm/civicrm-core/commit/da08fd08011964b6ad2187061a89f720af20106a), but it was a poor starting point; MD5 is not encryption, so what changes it does make are reversible. It also targeted a limited number of fields, so your DB would still contain identifying data even if not deanonymised ...
  * **civicrm\_obfuscate\_db.php** (based on `encryptDB.php` above) is another script I wrote several years back. It generates a [set of SQL commands to obfuscate your DB](https://github.com/xurizaemon/org.civicrm.contrib.anonymize/blob/obfuscate_db/obfuscate_db.php). It's SQL-based so relatively fast, generates incomprehensible names ("bggerns srrssbb"), and like all the above options is not a complete solution.
  * Another option would be to construct a test environment with test only data that simulates all situations you need to test, and keep expanding it as you identify new bugs. Let's be realistic, few CiviCRM sites have budget for that
  
  To recap: none of the above are complete working solutions for what you want, and the solution you need depends on your context. But please do contribute your experiences to the community - most of the above have a Github issue queue you could submit improvements / feedback to!
  
  One particular sticking point for several of the above seems to be CiviCRM's "Database Logging" tables, which by default use a format which complicates sanitisation. If you're testing or using any of the above, you should definitely review whether those tables are being de-identified completely in the process.
  
  > **Comments:**
  > **username_3**: <stands up and starts clapping>  
  > **username_1**: CRM-20346 just got closed.  
  > **username_0**: username_2 username_2, so much effort went into this answer. Really thankful for that! I will surely study the suggested options and give feedback to, in a way, "pay back" your effort.  
  > **username_2**: Thanks! You aren't "nully", previously of the FSF are you? Guessing not, but if so, I'd love to pick your brains on something.  
  > **username_4**: A word of warning - make sure that outgoing email is disabled!  You don't want any of your CiviRules etc. being triggered and sending out emails.  I saw a colleague who had [CiviCRM WordPress Profile Sync](https://wordpress.org/plugins/civicrm-wp-profile-sync/) installed, which meant that changing the email address in Civi changed the email in WordPress - which sent out a "your email address has been changed" message to everyone who had a login...  
  
  ---
  
  