Answers.Body: <p>How many emails are you trying to send? mailer cron=8 is really if you want to send a lot (I would not consider putting something else than 1 if less than 100k emails sent).</p>
  
  <p>As for the job size at 1000. you are on the lower side, and would probably put at least an order of magnitude higher.</p>
  
  <p>With more "normal" settings, you should get a better behaviour.</p>
  
  <p>This being said:
  1) yes, sending a mailing means a lot of traffic around your db, it's normal, nothing to worry about, it's big queries to record which emails have been sent</p>
  
  <p>2) yes, each email is sent a separate email, here again, it's normal and works as intended</p>
  
  <p>What is less normal is the number of connections. If you use smtp, it will create one connection and push all the emails of the batch in this connection. Obvioulsy the more cron jobs running you have, the more connection, hence my recommendation to lower it to 1 to see how it goes</p>
  
  <p>I'm not familiar with the sparkpost extension, it might use a http rest api, and depending how it's designed, make more requests, you should check the code or look directly with the authors of the extension.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T07:15:51.270Z
Answers.LastActivityDate: 2018-08-13T07:15:51.270Z
Answers.UserDisplayName: Xavier
Body: <p>Hopefully someone can help..
  First the particulars: </p>
  
  <p>CiviCRM 5.1.2 / Wordpress<br>
  Sparkpost Extension: 1.1<br>
  Mailer Batch: 0<br>
  Mailer throttle: 0<br>
  Mailer job: 1000<br>
  Mailer Cron: 8<br></p>
  
  <p>It appears that when sending out bulk emails, Civi is generating them one at a time (we turned off Smarty), which is impacting email performance.</p>
  
  <p><strong>How was I able to determine this ?</strong><br>
  - Doing "tail -f" on the civi log, I saw a copy of every email being recorded, one at a time. <br>
  - A constant stream of activity on the database, with queries (civicrm_relationship) and recording of the email activity in the database. When the mails were finished, the database activity ceased.<br>
  - And the clincher... a continuous stream of sockets being opened and timing out. Running "netstat -an" and grep-ing the ip, I saw a few hundred connections to the same ip (though we are only running a batch of 8), and only the ports changing. The connections would time out, and new ports would come in..this was a pattern observed throughout the life of the email process.</p>
  
  <p>Can someone please confirm that this is the expected behavior of Civi and if so, is there a way to batch emails. Also, is there a way to keep connections open ? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-09T19:55:47.547Z
Comments.Text: I'll add comments based on what we're finding (since I'm noticing that info is pretty sparse). We loaded the new Sparkpost extension and that seems to have fixed one issue where I was noticing tons of connections coming and going to  the SparkPost IPs.. The connections now seem to be persistent. Now that it's getting a little clearer, I'm noticing the same behavior to MySQL (RDS).. connections coming and going.. so it appears that CiviMail is opening a connection, doing some pretty ugly SQL queries and then closing the connection.. ie. the connections aren't persisting.
Comments.UserDisplayName: Michael LiVolsi
Tags: <email><sparkpost>
Title: CiviCRM Slow Bulk Emails to Sparkpost via Rest
ThreadText: # CiviCRM Slow Bulk Emails to Sparkpost via Rest
  
  (Asked by: username_0 on 2018-08-06)
  
  Hopefully someone can help..
  First the particulars:
  
  CiviCRM 5.1.2 / Wordpress  
  Sparkpost Extension: 1.1  
  Mailer Batch: 0  
  Mailer throttle: 0  
  Mailer job: 1000  
  Mailer Cron: 8
  
  It appears that when sending out bulk emails, Civi is generating them one at a time (we turned off Smarty), which is impacting email performance.
  
  **How was I able to determine this ?**  
  - Doing "tail -f" on the civi log, I saw a copy of every email being recorded, one at a time.   
  - A constant stream of activity on the database, with queries (civicrm\_relationship) and recording of the email activity in the database. When the mails were finished, the database activity ceased.  
  - And the clincher... a continuous stream of sockets being opened and timing out. Running "netstat -an" and grep-ing the ip, I saw a few hundred connections to the same ip (though we are only running a batch of 8), and only the ports changing. The connections would time out, and new ports would come in..this was a pattern observed throughout the life of the email process.
  
  Can someone please confirm that this is the expected behavior of Civi and if so, is there a way to batch emails. Also, is there a way to keep connections open ?
  
  **Tags:** email, sparkpost
  
  > **Comments:**
  > **username_0**: I'll add comments based on what we're finding (since I'm noticing that info is pretty sparse). We loaded the new Sparkpost extension and that seems to have fixed one issue where I was noticing tons of connections coming and going to  the SparkPost IPs.. The connections now seem to be persistent. Now that it's getting a little clearer, I'm noticing the same behavior to MySQL (RDS).. connections coming and going.. so it appears that CiviMail is opening a connection, doing some pretty ugly SQL queries and then closing the connection.. ie. the connections aren't persisting.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-13. Score: 1
  
  How many emails are you trying to send? mailer cron=8 is really if you want to send a lot (I would not consider putting something else than 1 if less than 100k emails sent).
  
  As for the job size at 1000. you are on the lower side, and would probably put at least an order of magnitude higher.
  
  With more "normal" settings, you should get a better behaviour.
  
  This being said:
  1) yes, sending a mailing means a lot of traffic around your db, it's normal, nothing to worry about, it's big queries to record which emails have been sent
  
  2) yes, each email is sent a separate email, here again, it's normal and works as intended
  
  What is less normal is the number of connections. If you use smtp, it will create one connection and push all the emails of the batch in this connection. Obvioulsy the more cron jobs running you have, the more connection, hence my recommendation to lower it to 1 to see how it goes
  
  I'm not familiar with the sparkpost extension, it might use a http rest api, and depending how it's designed, make more requests, you should check the code or look directly with the authors of the extension.
  
  
  ---
  
  