Answers.Body: <p>I've created an extension to handle bounces through Amazon SNS notifications, see <a href="https://github.com/mecachisenros/civicrm-ses" rel="nofollow noreferrer">civicrm-ses</a>.</p>
  
  <p>It exposes a webhook to process SNS notifications and <a href="https://github.com/mecachisenros/civicrm-ses/blob/master/README.md#usage" rel="nofollow noreferrer">map and create</a> bounces when appropriate.</p>
  
  <p>Hope you find it useful.</p>
  
  <p><strong>PS</strong>: although the extension is in <code>alpha</code> stage, I've been using it in production for a month (~60,000 emails processed), test in a dev environment first if you like :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-20T08:03:44.330Z
Answers.Text: Was on holiday, will check soon and give feedback! Thanks for the post!
Answers.UserDisplayName: Phil_B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T23:53:18.543Z
Answers.LastActivityDate: 2018-05-29T23:53:18.543Z
Answers.UserDisplayName: Andrei
Body: <p>I have a setup with CiviCRM and Amazon SES, everything is working fine.</p>
  
  <p>A Problem arises with the bounce handling. Amazon can either do that internally via Amazon SNS, or just forward the mails to your domain for common handling (Catchall or VERP).</p>
  
  <p>I wanted to use the SNS mechanism, and amazon sends the mail correctly to the specified address, but the email isn't a typical bounce mail, but instead the original headers are JSON encoded like this:</p>
  
  <pre><code>{   "notificationType": "Bounce",   "bounce": {
      "bounceType": "Permanent",
      "bounceSubType": "General",
      "bouncedRecipients": [{
        "emailAddress": "bounce@simulator.amazonses.com",
        "action": "failed",
        "status": "5.1.1",
        "diagnosticCode": "smtp; 550 5.1.1 user unknown"
      }],
      "timestamp": "2017-11-20T11:05:06.477Z",
      "feedbackId": "0102015fd91a6479-ae22395c-1f7d-4423-a9e5-e0715bffc731-000000",
      "remoteMtaIp": "1.2.3.4",
      "reportingMTA": "dsn; a4-8.smtp-out.eu-west-1.amazonses.com"   },   "mail": {
      "timestamp": "2017-11-20T11:05:04.000Z",
      "source": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com",
      "sourceArn": "ARN_INTERNAL:identity/example.com",
      "sourceIp": "1.2.3.4",
      "sendingAccountId": "__ID__",
      "messageId": "0102015fd91a5ca3-93c245fe-e333-46ae-8fed-96160d20cfab-000000",
      "destination": ["bounce@simulator.amazonses.com"],
      "headersTruncated": false,
      "headers": [{
        "name": "Received",
        "value": "from crm.example.com (mx02.example.com [1.2.3.4]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-2638500019) id MbY8b9mKuaex7LdPcwd8 for bounce@simulator.amazonses.com; Mon, 20 Nov 2017 11:05:04 +0000 (UTC)"
      }, {
        "name": "MIME-Version",
        "value": "1.0"
      }, {
        "name": "Content-Type",
        "value": "multipart/alternative; boundary=\"=_99c8dccfc484a99b0f6101141a34c4ae\""
      }, {
        "name": "Reply-To",
        "value": "\"Example Company\" &lt;info@example.com&gt;"
      }, {
        "name": "Return-Path",
        "value": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com"
      }, {
        "name": "From",
        "value": "\"Example Company\" &lt;info@example.com&gt;"
      }, {
        "name": "Subject",
        "value": "Bounce Test – Monday 4"
      }, {
        "name": "List-Unsubscribe",
        "value": "&lt;mailto:bounce.crm+u.83.52.2a7c92a4101dd5fd@example.com&gt;"
      }, {
        "name": "job_id",
        "value": "83"
      }, {
        "name": "To",
        "value": "Tom TestUser &lt;bounce@simulator.amazonses.com&gt;"
      }, {
        "name": "Precedence",
        "value": "bulk"
      }, {
        "name": "X-CiviMail-Bounce",
        "value": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com"
      }, {
        "name": "Date",
        "value": "Mon, 20 Nov 2017 12:05:04 +0100"
      }],
      "commonHeaders": {
        "returnPath": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com",
        "from": ["Example Company &lt;info@example.com&gt;"],
        "replyTo": ["Example Company &lt;info@example.com&gt;"],
        "date": "Mon, 20 Nov 2017 12:05:04 +0100",
        "to": ["Tom TestUser &lt;bounce@simulator.amazonses.com&gt;"],
        "subject": "Bounce Test – Monday 4"
      }   } }
  </code></pre>
  
  <p>Civi cannot handle this type of message (to the best of my knowledge) and it is then moved to the ignored folder.
  I'm pretty sure I am not the first one to have this problem, but I couldn't find any solutions to this. Maybe someone has written a parser for those type of messages already?</p>
  
  <p>Help/Suggestions are welcome! </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-20T14:26:10.383Z
Comments.Text: If you want Civi to handle the bounce directly, why not just use the email option in SES rather than SNS?
Comments.UserDisplayName: Aidan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-20T15:17:55.680Z
Comments.Text: doesn't work! I don't have a catchall/VERP handling, and I don't receive the original emails, only the JSON formatted ones.
  Might be that I have something misconfigured, but I'm pretty sure I tried all options!
Comments.UserDisplayName: Phil_B
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-19T07:25:58.180Z
Comments.Text: Did you have any joy with this in the end, Phil_B?
Comments.UserDisplayName: michaelmcandrew
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-19T07:44:17.120Z
Comments.Text: no, not really to be honest. I see 2 different approaches to this: Try and get the email option in SES to work properly - I think SES does some Magic in the background to get mails to that address, or introduce new patterns to CiviCRM for handling the JSON encoded bounces properly. https://github.com/systopia/de.systopia.mailingtools might help with that.
Comments.UserDisplayName: Phil_B
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-19T08:36:48.630Z
Comments.Text: Hey Phil, most likely I am going to write an extension that exposes an endpoint for subscribing to the AWS SNS and processes them appropriately. That is the method that other extensions (like sparkpost) use. Any thoughts / help very welcome.
Comments.UserDisplayName: michaelmcandrew
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-19T08:42:39.443Z
Comments.Text: Excellent! Let me know when/if we can provide some assistance, and we can figure something out!
Comments.UserDisplayName: Phil_B
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-19T09:06:03.153Z
Comments.Text: cool - will do :)
Comments.UserDisplayName: michaelmcandrew
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-07T11:27:34.483Z
Comments.Text: @michaelmcandrew I'm working on an extension to support bounce handling via SES/SNS, I'll keep you posted by the end of this week
Comments.UserDisplayName: Andrei
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-07T12:18:43.327Z
Comments.Text: @Andrei - ok cool - happy to help out - look forward to seeing it :)
Comments.UserDisplayName: michaelmcandrew
Tags: <email><bounce><amazon-ses>
Title: Bounce Handling with Amazon SES/SNS
ThreadText: # Bounce Handling with Amazon SES/SNS
  
  (Asked by: username_0 on 2017-11-20)
  
  I have a setup with CiviCRM and Amazon SES, everything is working fine.
  
  A Problem arises with the bounce handling. Amazon can either do that internally via Amazon SNS, or just forward the mails to your domain for common handling (Catchall or VERP).
  
  I wanted to use the SNS mechanism, and amazon sends the mail correctly to the specified address, but the email isn't a typical bounce mail, but instead the original headers are JSON encoded like this:
  
  ```
  {   "notificationType": "Bounce",   "bounce": {
      "bounceType": "Permanent",
      "bounceSubType": "General",
      "bouncedRecipients": [{
        "emailAddress": "bounce@simulator.amazonses.com",
        "action": "failed",
        "status": "5.1.1",
        "diagnosticCode": "smtp; 550 5.1.1 user unknown"
      }],
      "timestamp": "2017-11-20T11:05:06.477Z",
      "feedbackId": "0102015fd91a6479-ae22395c-1f7d-4423-a9e5-e0715bffc731-000000",
      "remoteMtaIp": "1.2.3.4",
      "reportingMTA": "dsn; a4-8.smtp-out.eu-west-1.amazonses.com"   },   "mail": {
      "timestamp": "2017-11-20T11:05:04.000Z",
      "source": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com",
      "sourceArn": "ARN_INTERNAL:identity/example.com",
      "sourceIp": "1.2.3.4",
      "sendingAccountId": "__ID__",
      "messageId": "0102015fd91a5ca3-93c245fe-e333-46ae-8fed-96160d20cfab-000000",
      "destination": ["bounce@simulator.amazonses.com"],
      "headersTruncated": false,
      "headers": [{
        "name": "Received",
        "value": "from crm.example.com (mx02.example.com [1.2.3.4]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-2638500019) id MbY8b9mKuaex7LdPcwd8 for bounce@simulator.amazonses.com; Mon, 20 Nov 2017 11:05:04 +0000 (UTC)"
      }, {
        "name": "MIME-Version",
        "value": "1.0"
      }, {
        "name": "Content-Type",
        "value": "multipart/alternative; boundary=\"=_99c8dccfc484a99b0f6101141a34c4ae\""
      }, {
        "name": "Reply-To",
        "value": "\"Example Company\" <info@example.com>"
      }, {
        "name": "Return-Path",
        "value": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com"
      }, {
        "name": "From",
        "value": "\"Example Company\" <info@example.com>"
      }, {
        "name": "Subject",
        "value": "Bounce Test – Monday 4"
      }, {
        "name": "List-Unsubscribe",
        "value": "<mailto:bounce.crm+u.83.52.2a7c92a4101dd5fd@example.com>"
      }, {
        "name": "job_id",
        "value": "83"
      }, {
        "name": "To",
        "value": "Tom TestUser <bounce@simulator.amazonses.com>"
      }, {
        "name": "Precedence",
        "value": "bulk"
      }, {
        "name": "X-CiviMail-Bounce",
        "value": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com"
      }, {
        "name": "Date",
        "value": "Mon, 20 Nov 2017 12:05:04 +0100"
      }],
      "commonHeaders": {
        "returnPath": "bounce.crm+b.83.52.2a7c92a4101dd5fd@example.com",
        "from": ["Example Company <info@example.com>"],
        "replyTo": ["Example Company <info@example.com>"],
        "date": "Mon, 20 Nov 2017 12:05:04 +0100",
        "to": ["Tom TestUser <bounce@simulator.amazonses.com>"],
        "subject": "Bounce Test – Monday 4"
      }   } }
  
  ```
  
  Civi cannot handle this type of message (to the best of my knowledge) and it is then moved to the ignored folder.
  I'm pretty sure I am not the first one to have this problem, but I couldn't find any solutions to this. Maybe someone has written a parser for those type of messages already?
  
  Help/Suggestions are welcome!
  
  **Tags:** email, bounce, amazon-ses
  
  > **Comments:**
  > **username_1**: If you want Civi to handle the bounce directly, why not just use the email option in SES rather than SNS?  
  > **username_0**: doesn't work! I don't have a catchall/VERP handling, and I don't receive the original emails, only the JSON formatted ones. Might be that I have something misconfigured, but I'm pretty sure I tried all options!  
  > **username_2**: Did you have any joy with this in the end, username_0?  
  > **username_0**: no, not really to be honest. I see 2 different approaches to this: Try and get the email option in SES to work properly - I think SES does some Magic in the background to get mails to that address, or introduce new patterns to CiviCRM for handling the JSON encoded bounces properly. https://github.com/systopia/de.systopia.mailingtools might help with that.  
  > **username_2**: Hey Phil, most likely I am going to write an extension that exposes an endpoint for subscribing to the AWS SNS and processes them appropriately. That is the method that other extensions (like sparkpost) use. Any thoughts / help very welcome.  
  > **username_0**: Excellent! Let me know when/if we can provide some assistance, and we can figure something out!  
  > **username_2**: cool - will do :)  
  > **username_3**: @username_2 I'm working on an extension to support bounce handling via SES/SNS, I'll keep you posted by the end of this week  
  > **username_2**: @username_3 - ok cool - happy to help out - look forward to seeing it :)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-05-29. Score: 3
  
  I've created an extension to handle bounces through Amazon SNS notifications, see [civicrm-ses](https://github.com/mecachisenros/civicrm-ses).
  
  It exposes a webhook to process SNS notifications and [map and create](https://github.com/mecachisenros/civicrm-ses/blob/master/README.md#usage) bounces when appropriate.
  
  Hope you find it useful.
  
  **PS**: although the extension is in `alpha` stage, I've been using it in production for a month (~60,000 emails processed), test in a dev environment first if you like :)
  
  > **Comments:**
  > **username_0**: Was on holiday, will check soon and give feedback! Thanks for the post!  
  
  ---
  
  