Answers.Body: <p>For those looking for a workaround for this issue, we created a tiny extension called <a href="https://github.com/greenpeace-cee/at.greenpeace.multibite" rel="nofollow noreferrer">Multibite</a> which replaces unsupported multi-byte characters with the Unicode replacement character (or any other character, depending on configuration). We did not want to pollute core with this code as the goal should be to <a href="https://lab.civicrm.org/dev/core/issues/339" rel="nofollow noreferrer">support utf8mb4</a>, after which the extension becomes obsolete.</p>
  
  <p>I should note that we aren't running this in production yet, so it hasn't seen much real-world testing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T13:53:19.627Z
Answers.Text: Just a note that you can change your database tables now to utf8mb4 and it doesn't seem to break anything in core and it works for new inbound emails going forward. But I never found a good way to retrieve the lost data and get it into civi.
Answers.UserDisplayName: Demerit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-14T18:24:33.960Z
Answers.Text: @Demerit that's definitely a workable approach that could be the better option for some or perhaps even most sites! My consideration was to keep our installations easily reproducible (we'd have to ship the charset change as an extension) and to avoid potential bugs with Civi's use of temporary tables and similar (Civi tends to create them ad-hoc for some operations. They mostly seem to store IDs, but if there are any with varchar columns, and the charset is explicitly set by core, I'd expect to see issues when unsupported multi-byte characters are involved. (Applying PR #13633 would work too.)
Answers.UserDisplayName: pfigel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T13:35:14.627Z
Answers.LastActivityDate: 2019-09-13T13:35:14.627Z
Answers.UserDisplayName: pfigel
Body: <p>A related question is <a href="https://civicrm.stackexchange.com/questions/15872/has-anyone-used-emojis-in-civimail-subject-body/15875#15875">Has anyone used Emojis in CiviMail subject/body?</a> and the answer is to change the database to use utf8mb4 which is going to be standard in CiviCRM in upcoming versions, but using utf8 currently I'm seeing that depending on the database (e.g. mysql vs. mariadb) and version (e.g. mysql 5.5 vs 5.7) it can fail noisily or can fail silently and just truncates anything after the emoji. So this is partly a PSA in case others haven't noticed silent fails.</p>
  
  <p>I was able to run this grep on the sites/default/files/civicrm/custom/CiviMail.processed folder (drupal - adjust as needed) to locate at least some candidates that have been truncated, but I doubt there's a way to then automate updating the activities in civi. If I find one will post back here.</p>
  
  <p><code>grep -r "=F[0-7]=[0-9A-F][0-9A-F]=[0-9A-F][0-9A-F]=[0-9A-F][0-9A-F]" sites/default/files/civicrm/custom/CiviMail.processed</code></p>
  
Tags: <email><activities>
Title: Email processor silently truncating anything after an emoji in inbound emails
ThreadText: # Email processor silently truncating anything after an emoji in inbound emails
  
  (Asked by: username_0 on 2019-05-11)
  
  A related question is [Has anyone used Emojis in CiviMail subject/body?](https://civicrm.stackexchange.com/questions/15872/has-anyone-used-emojis-in-civimail-subject-body/15875#15875) and the answer is to change the database to use utf8mb4 which is going to be standard in CiviCRM in upcoming versions, but using utf8 currently I'm seeing that depending on the database (e.g. mysql vs. mariadb) and version (e.g. mysql 5.5 vs 5.7) it can fail noisily or can fail silently and just truncates anything after the emoji. So this is partly a PSA in case others haven't noticed silent fails.
  
  I was able to run this grep on the sites/default/files/civicrm/custom/CiviMail.processed folder (drupal - adjust as needed) to locate at least some candidates that have been truncated, but I doubt there's a way to then automate updating the activities in civi. If I find one will post back here.
  
  `grep -r "=F[0-7]=[0-9A-F][0-9A-F]=[0-9A-F][0-9A-F]=[0-9A-F][0-9A-F]" sites/default/files/civicrm/custom/CiviMail.processed`
  
  **Tags:** email, activities
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-13. Score: 2
  
  For those looking for a workaround for this issue, we created a tiny extension called [Multibite](https://github.com/greenpeace-cee/at.greenpeace.multibite) which replaces unsupported multi-byte characters with the Unicode replacement character (or any other character, depending on configuration). We did not want to pollute core with this code as the goal should be to [support utf8mb4](https://lab.civicrm.org/dev/core/issues/339), after which the extension becomes obsolete.
  
  I should note that we aren't running this in production yet, so it hasn't seen much real-world testing.
  
  > **Comments:**
  > **username_0**: Just a note that you can change your database tables now to utf8mb4 and it doesn't seem to break anything in core and it works for new inbound emails going forward. But I never found a good way to retrieve the lost data and get it into civi.  
  > **username_1**: @username_0 that's definitely a workable approach that could be the better option for some or perhaps even most sites! My consideration was to keep our installations easily reproducible (we'd have to ship the charset change as an extension) and to avoid potential bugs with Civi's use of temporary tables and similar (Civi tends to create them ad-hoc for some operations. They mostly seem to store IDs, but if there are any with varchar columns, and the charset is explicitly set by core, I'd expect to see issues when unsupported multi-byte characters are involved. (Applying PR #13633 would work too.)  
  
  ---
  
  