Answers.Body: <p>I have found the workaround for my problem. 
  If you change the setting in my CiviCRM instance to send mailings in batches, by setting Mailer Batch Limit (to let's say 100 mailings every one or two minutes) this problem disappears. </p>
  
  <p>I guess that system gets overwhelmed when trying to process n URLs in couple of thousands emails, at the same time. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-21T15:19:37.337Z
Answers.LastActivityDate: 2018-02-21T15:19:37.337Z
Answers.UserDisplayName: Marko
Body: <p>I have a problem where my mailing is being sent crazy slow, like hundred emails an hour. At the same time my cpu utilization is of the charts where apache2 is using like 99% of the cpu. In the php error.log I'm getting the message for UrlFilter.php repeating for the line 62,63,64,65,and 68.</p>
  
  <pre><code>Maximum execution time of 30 seconds exceeded in /var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/UrlFilter.php on line 62
  </code></pre>
  
  <p>These are the affected lines:</p>
  
  <pre><code>    62 $htmls = preg_replace_callback(';(\&lt;img [^&gt;]*src *= *")([^"&gt;]+)(");i', $callback, $htmls);
      63 $htmls = preg_replace_callback(';(\&lt;img [^&gt;]*src *= *\')([^"&gt;]+)(\');i', $callback, $htmls);
      64 $htmls = preg_replace_callback(';(\&lt;table [^&gt;]*background *= *")([^"&gt;]+)(");i', $callback, $htmls);
      65 $htmls = preg_replace_callback(';(\&lt;table [^&gt;]*background *= *")([^\'&gt;]+)(\');i', $callback, $htmls);
      66 // WISHLIST: CSS backgrounds?
      67 return $htmls;
      68 }
  </code></pre>
  
  <p>My CiviCRM log gets flooded with the following message:</p>
  
  <pre><code>Jan 18 15:30:08  [info] 
  $Fatal Error Details = array(3) {
    ["message"]=&gt;
    string(116) "The path is a directory: /var/www/mysite/sites/default/files/civicrm/persist/contribute/images/uploads/static/"
    ["code"]=&gt;
    NULL
    ["exception"]=&gt;
    object(ImagickException)#114 (7) {
      ["message":protected]=&gt;
      string(116) "The path is a directory: /var/www/mysite/sites/default/files/civicrm/persist/contribute/images/uploads/static/"
      ["string":"Exception":private]=&gt;
      string(0) ""
      ["code":protected]=&gt;
      int(1)
      ["file":protected]=&gt;
      string(104) "/var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/Utils.php"
      ["line":protected]=&gt;
      int(514)
      ["trace":"Exception":private]=&gt;
      array(10) {
        [0]=&gt;
        array(6) {
          ["file"]=&gt;
          string(104) "/var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/Utils.php"
          ["line"]=&gt;
          int(514)
          ["function"]=&gt;
          string(11) "__construct"
          ["class"]=&gt;
          string(7) "Imagick"
          ["type"]=&gt;
          string(2) "-&gt;"
          ["args"]=&gt;
          array(1) {
            [0]=&gt;
            string(91) "/var/www/mysite/sites/default/files/civicrm/persist/contribute/images/uploads/static/"
          }
        }
        [1]=&gt;
        array(6) {
          ["file"]=&gt;
          string(104) "/var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/Utils.php"
          ["line"]=&gt;
          int(324)
          ["function"]=&gt;
          string(11) "resizeImage"
          ["class"]=&gt;
          string(17) "CRM_Mosaico_Utils"
          ["type"]=&gt;
          string(2) "::"
          ["args"]=&gt;
          array(4) {
            [0]=&gt;
            string(0) ""
            [1]=&gt;
            NULL
            [2]=&gt;
            int(0)
            [3]=&gt;
            int(0)
          }
        }
        [2]=&gt;
        array(4) {
          ["function"]=&gt;
          string(10) "processImg"
          ["class"]=&gt;
          string(17) "CRM_Mosaico_Utils"
          ["type"]=&gt;
          string(2) "::"
          ["args"]=&gt;
          array(0) {
          }
        }
        [3]=&gt;
        array(4) {
          ["file"]=&gt;
          string(67) "/var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php"
          ["line"]=&gt;
          int(275)
          ["function"]=&gt;
          string(14) "call_user_func"
          ["args"]=&gt;
          array(1) {
            [0]=&gt;
            array(2) {
              [0]=&gt;
              string(17) "CRM_Mosaico_Utils"
              [1]=&gt;
              string(10) "processImg"
            }
          }
        }
        [4]=&gt;
        array(6) {
          ["file"]=&gt;
          string(67) "/var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php"
          ["line"]=&gt;
          int(84)
          ["function"]=&gt;
          string(7) "runItem"
          ["class"]=&gt;
          string(15) "CRM_Core_Invoke"
          ["type"]=&gt;
          string(2) "::"
          ["args"]=&gt;
          array(1) {
            [0]=&gt;
            array(14) {
              ["id"]=&gt;
              string(3) "443"
              ["domain_id"]=&gt;
              string(1) "1"
              ["path"]=&gt;
              string(19) "civicrm/mosaico/img"
              ["title"]=&gt;
              string(24) "Integration with Mosaico"
              ["access_callback"]=&gt;
              string(1) "1"
              ["access_arguments"]=&gt;
              array(2) {
                [0]=&gt;
                array(1) {
                  [0]=&gt;
                  string(14) "access CiviCRM"
                }
                [1]=&gt;
                string(3) "and"
              }
              ["page_callback"]=&gt;
              array(2) {
                [0]=&gt;
                string(17) "CRM_Mosaico_Utils"
                [1]=&gt;
                string(10) "processImg"
              }
              ["breadcrumb"]=&gt;
              array(1) {
                [0]=&gt;
                array(2) {
                  ["title"]=&gt;
                  string(7) "CiviCRM"
                  ["url"]=&gt;
                  string(16) "/civicrm?reset=1"
                }
              }
              ["is_public"]=&gt;
              string(1) "1"
              ["is_ssl"]=&gt;
              string(1) "0"
              ["weight"]=&gt;
              string(1) "1"
              ["type"]=&gt;
              string(1) "1"
              ["page_type"]=&gt;
              string(1) "0"
              ["page_arguments"]=&gt;
              bool(false)
            }
          }
        }
        [5]=&gt;
        array(6) {
          ["file"]=&gt;
          string(67) "/var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php"
          ["line"]=&gt;
          int(52)
          ["function"]=&gt;
          string(7) "_invoke"
          ["class"]=&gt;
          string(15) "CRM_Core_Invoke"
          ["type"]=&gt;
          string(2) "::"
          ["args"]=&gt;
          array(1) {
            [0]=&gt;
            array(3) {
              [0]=&gt;
              string(7) "civicrm"
              [1]=&gt;
              string(7) "mosaico"
              [2]=&gt;
              string(3) "img"
            }
          }
        }
        [6]=&gt;
        array(6) {
          ["file"]=&gt;
          string(69) "/var/www/mysite/sites/all/modules/civicrm/drupal/civicrm.module"
          ["line"]=&gt;
          int(448)
          ["function"]=&gt;
          string(6) "invoke"
          ["class"]=&gt;
          string(15) "CRM_Core_Invoke"
          ["type"]=&gt;
          string(2) "::"
          ["args"]=&gt;
          array(1) {
            [0]=&gt;
            &amp;array(3) {
              [0]=&gt;
              string(7) "civicrm"
              [1]=&gt;
              string(7) "mosaico"
              [2]=&gt;
              string(3) "img"
            }
          }
        }
        [7]=&gt;
        array(2) {
          ["function"]=&gt;
          string(14) "civicrm_invoke"
          ["args"]=&gt;
          array(2) {
            [0]=&gt;
            &amp;string(7) "mosaico"
            [1]=&gt;
            &amp;string(3) "img"
          }
        }
        [8]=&gt;
        array(4) {
          ["file"]=&gt;
          string(39) "/var/www/mysite/includes/menu.inc"
          ["line"]=&gt;
          int(527)
          ["function"]=&gt;
          string(20) "call_user_func_array"
          ["args"]=&gt;
          array(2) {
            [0]=&gt;
            &amp;string(14) "civicrm_invoke"
            [1]=&gt;
            &amp;array(2) {
              [0]=&gt;
              string(7) "mosaico"
              [1]=&gt;
              string(3) "img"
            }
          }
        }
        [9]=&gt;
        array(4) {
          ["file"]=&gt;
          string(31) "/var/www/mysite/index.php"
          ["line"]=&gt;
          int(21)
          ["function"]=&gt;
          string(27) "menu_execute_active_handler"
          ["args"]=&gt;
          array(0) {
          }
        }
      }
      ["previous":"Exception":private]=&gt;
      NULL
    }
  }
  
  
  Jan 18 15:30:08  [info] $backTrace = #0 /var/www/mysite/sites/all/modules/civicrm/CRM/Core/Error.php(456): CRM_Core_Error::backtrace("backTrace", TRUE)
  #1 /var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php(55): CRM_Core_Error::handleUnhandledException(Object(ImagickException))
  #2 /var/www/mysite/sites/all/modules/civicrm/drupal/civicrm.module(448): CRM_Core_Invoke::invoke((Array:3))
  #3 [internal function](): civicrm_invoke("mosaico", "img")
  #4 /var/www/mysite/includes/menu.inc(527): call_user_func_array("civicrm_invoke", (Array:2))
  #5 /var/www/mysite/index.php(21): menu_execute_active_handler()
  #6 {main}
  </code></pre>
  
  <p>Restarting apache fixes the problem temporarily. The mails get sent as they should the errors stop, but the problem is reoccurring couple of weeks when I'm sending my next mailing. 
  Is my system missconfigured in some way?</p>
  
  <p>I'm using CiviCRM 4.7.27 on Drupal 7.56 with <a href="https://github.com/mattwire/uk.co.vedaconsulting.mosaico/tree/2.1-mjwconsult1" rel="nofollow noreferrer">Matt Wire's Mosaico 2.0-beta3-mjwconsult3</a> </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-19T15:44:12.537Z
Comments.Text: What's the output of var_dump($htmls) before it is returned? Imagick throws an error because there is problem with image patch.
Comments.UserDisplayName: Djordje Marjanovic
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-22T11:16:09.080Z
Comments.Text: I can't recreate the error without sending the mailing (to couple of thousands of addresses) so I can't experiment now. I might do something when we send our next mailing in a couple of days so I'll report then.
Comments.UserDisplayName: Marko
Tags: <civimail><email><php><mailing><mosaico>
Title: Very slow mailing job with Mosaico with Imagick and UrlFilter.php errors
ThreadText: # Very slow mailing job with Mosaico with Imagick and UrlFilter.php errors
  
  (Asked by: username_0 on 2018-01-18)
  
  I have a problem where my mailing is being sent crazy slow, like hundred emails an hour. At the same time my cpu utilization is of the charts where apache2 is using like 99% of the cpu. In the php error.log I'm getting the message for UrlFilter.php repeating for the line 62,63,64,65,and 68.
  Maximum execution time of 30 seconds exceeded in /var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/UrlFilter.php on line 62
  
  These are the affected lines:
      62 $htmls = preg_replace_callback(';(\]*src *= *")([^">]+)(");i', $callback, $htmls);
      63 $htmls = preg_replace_callback(';(\]*src *= *\')([^">]+)(\');i', $callback, $htmls);
      64 $htmls = preg_replace_callback(';(\]*background *= *")([^">]+)(");i', $callback, $htmls);
      65 $htmls = preg_replace_callback(';(\]*background *= *")([^\'>]+)(\');i', $callback, $htmls);
      66 // WISHLIST: CSS backgrounds?
      67 return $htmls;
      68 }
  
  My CiviCRM log gets flooded with the following message:
  Jan 18 15:30:08  [info] 
  $Fatal Error Details = array(3) {
    ["message"]=>
    string(116) "The path is a directory: /var/www/mysite/sites/default/files/civicrm/persist/contribute/images/uploads/static/"
    ["code"]=>
    NULL
    ["exception"]=>
    object(ImagickException)#114 (7) {
      ["message":protected]=>
      string(116) "The path is a directory: /var/www/mysite/sites/default/files/civicrm/persist/contribute/images/uploads/static/"
      ["string":"Exception":private]=>
      string(0) ""
      ["code":protected]=>
      int(1)
      ["file":protected]=>
      string(104) "/var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/Utils.php"
      ["line":protected]=>
      int(514)
      ["trace":"Exception":private]=>
      array(10) {
        [0]=>
        array(6) {
          ["file"]=>
          string(104) "/var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/Utils.php"
          ["line"]=>
          int(514)
          ["function"]=>
          string(11) "__construct"
          ["class"]=>
          string(7) "Imagick"
          ["type"]=>
          string(2) "->"
          ["args"]=>
          array(1) {
            [0]=>
            string(91) "/var/www/mysite/sites/default/files/civicrm/persist/contribute/images/uploads/static/"
          }
        }
        [1]=>
        array(6) {
          ["file"]=>
          string(104) "/var/www/mysite/sites/default/files/civicrm/ext/uk.co.vedaconsulting.mosaico/CRM/Mosaico/Utils.php"
          ["line"]=>
          int(324)
          ["function"]=>
          string(11) "resizeImage"
          ["class"]=>
          string(17) "CRM_Mosaico_Utils"
          ["type"]=>
          string(2) "::"
          ["args"]=>
          array(4) {
            [0]=>
            string(0) ""
            [1]=>
            NULL
            [2]=>
            int(0)
            [3]=>
            int(0)
          }
        }
        [2]=>
        array(4) {
          ["function"]=>
          string(10) "processImg"
          ["class"]=>
          string(17) "CRM_Mosaico_Utils"
          ["type"]=>
          string(2) "::"
          ["args"]=>
          array(0) {
          }
        }
        [3]=>
        array(4) {
          ["file"]=>
          string(67) "/var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php"
          ["line"]=>
          int(275)
          ["function"]=>
          string(14) "call_user_func"
          ["args"]=>
          array(1) {
            [0]=>
            array(2) {
              [0]=>
              string(17) "CRM_Mosaico_Utils"
              [1]=>
              string(10) "processImg"
            }
          }
        }
        [4]=>
        array(6) {
          ["file"]=>
          string(67) "/var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php"
          ["line"]=>
          int(84)
          ["function"]=>
          string(7) "runItem"
          ["class"]=>
          string(15) "CRM_Core_Invoke"
          ["type"]=>
          string(2) "::"
          ["args"]=>
          array(1) {
            [0]=>
            array(14) {
              ["id"]=>
              string(3) "443"
              ["domain_id"]=>
              string(1) "1"
              ["path"]=>
              string(19) "civicrm/mosaico/img"
              ["title"]=>
              string(24) "Integration with Mosaico"
              ["access_callback"]=>
              string(1) "1"
              ["access_arguments"]=>
              array(2) {
                [0]=>
                array(1) {
                  [0]=>
                  string(14) "access CiviCRM"
                }
                [1]=>
                string(3) "and"
              }
              ["page_callback"]=>
              array(2) {
                [0]=>
                string(17) "CRM_Mosaico_Utils"
                [1]=>
                string(10) "processImg"
              }
              ["breadcrumb"]=>
              array(1) {
                [0]=>
                array(2) {
                  ["title"]=>
                  string(7) "CiviCRM"
                  ["url"]=>
                  string(16) "/civicrm?reset=1"
                }
              }
              ["is_public"]=>
              string(1) "1"
              ["is_ssl"]=>
              string(1) "0"
              ["weight"]=>
              string(1) "1"
              ["type"]=>
              string(1) "1"
              ["page_type"]=>
              string(1) "0"
              ["page_arguments"]=>
              bool(false)
            }
          }
        }
        [5]=>
        array(6) {
          ["file"]=>
          string(67) "/var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php"
          ["line"]=>
          int(52)
          ["function"]=>
          string(7) "_invoke"
          ["class"]=>
          string(15) "CRM_Core_Invoke"
          ["type"]=>
          string(2) "::"
          ["args"]=>
          array(1) {
            [0]=>
            array(3) {
              [0]=>
              string(7) "civicrm"
              [1]=>
              string(7) "mosaico"
              [2]=>
              string(3) "img"
            }
          }
        }
        [6]=>
        array(6) {
          ["file"]=>
          string(69) "/var/www/mysite/sites/all/modules/civicrm/drupal/civicrm.module"
          ["line"]=>
          int(448)
          ["function"]=>
          string(6) "invoke"
          ["class"]=>
          string(15) "CRM_Core_Invoke"
          ["type"]=>
          string(2) "::"
          ["args"]=>
          array(1) {
            [0]=>
            &array(3) {
              [0]=>
              string(7) "civicrm"
              [1]=>
              string(7) "mosaico"
              [2]=>
              string(3) "img"
            }
          }
        }
        [7]=>
        array(2) {
          ["function"]=>
          string(14) "civicrm_invoke"
          ["args"]=>
          array(2) {
            [0]=>
            &string(7) "mosaico"
            [1]=>
            &string(3) "img"
          }
        }
        [8]=>
        array(4) {
          ["file"]=>
          string(39) "/var/www/mysite/includes/menu.inc"
          ["line"]=>
          int(527)
          ["function"]=>
          string(20) "call_user_func_array"
          ["args"]=>
          array(2) {
            [0]=>
            &string(14) "civicrm_invoke"
            [1]=>
            &array(2) {
              [0]=>
              string(7) "mosaico"
              [1]=>
              string(3) "img"
            }
          }
        }
        [9]=>
        array(4) {
          ["file"]=>
          string(31) "/var/www/mysite/index.php"
          ["line"]=>
          int(21)
          ["function"]=>
          string(27) "menu_execute_active_handler"
          ["args"]=>
          array(0) {
          }
        }
      }
      ["previous":"Exception":private]=>
      NULL
    }
  }
  
  
  Jan 18 15:30:08  [info] $backTrace = #0 /var/www/mysite/sites/all/modules/civicrm/CRM/Core/Error.php(456): CRM_Core_Error::backtrace("backTrace", TRUE)
  #1 /var/www/mysite/sites/all/modules/civicrm/CRM/Core/Invoke.php(55): CRM_Core_Error::handleUnhandledException(Object(ImagickException))
  #2 /var/www/mysite/sites/all/modules/civicrm/drupal/civicrm.module(448): CRM_Core_Invoke::invoke((Array:3))
  #3 [internal function](): civicrm_invoke("mosaico", "img")
  #4 /var/www/mysite/includes/menu.inc(527): call_user_func_array("civicrm_invoke", (Array:2))
  #5 /var/www/mysite/index.php(21): menu_execute_active_handler()
  #6 {main}
  
  Restarting apache fixes the problem temporarily. The mails get sent as they should the errors stop, but the problem is reoccurring couple of weeks when I'm sending my next mailing. 
  Is my system missconfigured in some way?
  I'm using CiviCRM 4.7.27 on Drupal 7.56 with Matt Wire's Mosaico 2.0-beta3-mjwconsult3
  
  **Tags:** civimail, email, php, mailing, mosaico
  
  > **Comments:**
  > **username_1**: What's the output of var_dump($htmls) before it is returned? Imagick throws an error because there is problem with image patch.  
  > **username_0**: I can't recreate the error without sending the mailing (to couple of thousands of addresses) so I can't experiment now. I might do something when we send our next mailing in a couple of days so I'll report then.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-02-21. Score: 0 | Accepted answer
  
  I have found the workaround for my problem.
  If you change the setting in my CiviCRM instance to send mailings in batches, by setting Mailer Batch Limit (to let's say 100 mailings every one or two minutes) this problem disappears.
  
  I guess that system gets overwhelmed when trying to process n URLs in couple of thousands emails, at the same time.
  
  
  ---
  
  