Body: <p>I am working on a background job in a custom plugin.
  On my local Laravel Valet setup it runs fine until it runs out of time.
  On my VPS which has same php version and memory settings (and mysql db) the queue fails almost instantly with a memory issue:</p>
  <pre><code>The command &quot;'/opt/cpanel/ea-php74/root/usr/bin/php' 'craft' 'queue/exec' '247' '3600' '1' '30302' '--color='&quot; failed. Exit Code: 255(Unknown error) Working directory: /home/craft 
  Output: ================ Error Output: ================ 
  PHP Fatal Error 'yii\base\ErrorException' with message 'Allowed memory size of 268435456 bytes exhausted (tried to allocate 12288 bytes)' in /home/craft/vendor/craftcms/cms/src/elements/db/ElementQuery.php:1906 
  Stack trace: #0 [internal function]: yii\base\ErrorHandler-&gt;handleFatalError() #1 {main}
  </code></pre>
  <p>Below is my job code. The number of entries to count is approx 9000. The job is meant to loop thru them and add some data from a 3rd party API.</p>
  <p>Questions:</p>
  <ol>
  <li>Why is it failing instantly on VPS but running locally ok. THe VPS has more memory than my mac</li>
  <li>Do i need to paginate the results and then loop thru them in batches?</li>
  </ol>
  <p>I am using the async plugin assuming this would help.</p>
  <p>Any thoughts welcome</p>
  <pre><code>    public function execute($queue)
      {
  
  
          // Do work here
          $query = Entry::find();
          if (!empty($this-&gt;criteria)) {
              Craft::configure($query, $this-&gt;criteria);
          }
          $query
              -&gt;section('imdData')
              -&gt;orderBy('id')
              -&gt;limit(null)
              -&gt;all();
  
          $totalElements = $query-&gt;count();
          $currentElement = 0;
  
          try {
              foreach (\craft\helpers\Db::each($query) as $element) {
  
  
                  $this-&gt;setProgress($queue, $currentElement++ / $totalElements);
                  $match = Entry::find()-&gt;section('mapitOlfData')-&gt;ons($element-&gt;lsoacode)-&gt;one(); //one because we want one entry returned and lsoanames are unique
  
                  if ($match) {
                      // if we have a match then call the mapit api with the area id
                      $geoData = $this-&gt;fetchGeoJson($match-&gt;mapitAreaId);
  
                      $element-&gt;setFieldValue('mapitAreaId', $match-&gt;mapitAreaId);
                      $element-&gt;setFieldValue('geojson', $geoData);
  
                      if (!Craft::$app-&gt;getElements()-&gt;saveElement($element)) {
                          Craft::error(
                              ' || Couldnt save funding entry: ' . $element-&gt;id . ':' . $element-&gt;title . '| lsoa:' . $element-&gt;lsoacode,
                              __METHOD__
                          );
                      }else{
                          Craft::debug(
                              Craft::t(
                                  'mapitapi',
                                  'Sleeping for 2 secs ..saved funding entry: ' . $element-&gt;id . ':' . $element-&gt;title . '| lsoa:' . $element-&gt;lsoacode
                              ),
                              __METHOD__
                          );
                          sleep(1);
                      }
                  }
  
                  //}
              }
          } catch (Exception $e) {
              echo '[error]: '
                  . $e-&gt;getMessage()
                  . 'exception while processing '
                  . $currentElement . '/' . $totalElements
                  . ' - processing asset: ' . $element-&gt;title
                  . ' from field: ' . $element-&gt;lsoacode . PHP_EOL;
  
          }
      }
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-06T03:13:59.317Z
Comments.Text: The VPC might have more system memory, but your local one might have more assigned to PHP. What is your php.ini file's `memory_limit` setting set to locally? It looks like it's 256M on the VPC.
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-06T07:29:06.570Z
Comments.Text: Locally memory_limit is 512M, on VPS I set it to 512M and 1024M with the same result
Comments.UserDisplayName: joomkit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-06T20:41:18.970Z
Comments.Text: @joomkit:  stupid question, did you edit the right config file? After you raised the memory limit, did the error message show the increased memory limit? Looks like you're running cpanel and easy apache? The CLI config is usually different than the web server one, not sure if this [doc](https://docs.cpanel.net/ea4/php/easyapache4-and-the-ea-php-cli-package/) is helpful... it's also possible your host disabled config editing...
Comments.UserDisplayName: RitterKnight
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-06T23:00:04.143Z
Comments.Text: @RitterKnight fair question!. Yes and the chnage shows in craft php inspection panel. However I have better coding now which seems to have bypassed the issue. See https://craftcms.stackexchange.com/questions/38733/how-to-batch-a-large-queue-job/38734#38734
Comments.UserDisplayName: joomkit
Tags: <plugin-development><queue>
Title: Why is my queue in custom plugin failing on VPS but not locally?
ThreadText: # Why is my queue in custom plugin failing on VPS but not locally?
  
  (Asked by: username_0 on 2021-12-05)
  
  I am working on a background job in a custom plugin.
  On my local Laravel Valet setup it runs fine until it runs out of time.
  On my VPS which has same php version and memory settings (and mysql db) the queue fails almost instantly with a memory issue:
  
  ```
  The command "'/opt/cpanel/ea-php74/root/usr/bin/php' 'craft' 'queue/exec' '247' '3600' '1' '30302' '--color='" failed. Exit Code: 255(Unknown error) Working directory: /home/craft 
  Output: ================ Error Output: ================ 
  PHP Fatal Error 'yii\base\ErrorException' with message 'Allowed memory size of 268435456 bytes exhausted (tried to allocate 12288 bytes)' in /home/craft/vendor/craftcms/cms/src/elements/db/ElementQuery.php:1906 
  Stack trace: #0 [internal function]: yii\base\ErrorHandler->handleFatalError() #1 {main}
  
  ```
  
  Below is my job code. The number of entries to count is approx 9000. The job is meant to loop thru them and add some data from a 3rd party API.
  
  Questions:
  
  1. Why is it failing instantly on VPS but running locally ok. THe VPS has more memory than my mac
  2. Do i need to paginate the results and then loop thru them in batches?
  
  I am using the async plugin assuming this would help.
  
  Any thoughts welcome
  
  ```
      public function execute($queue)
      {
  
          // Do work here
          $query = Entry::find();
          if (!empty($this->criteria)) {
              Craft::configure($query, $this->criteria);
          }
          $query
              ->section('imdData')
              ->orderBy('id')
              ->limit(null)
              ->all();
  
          $totalElements = $query->count();
          $currentElement = 0;
  
          try {
              foreach (\craft\helpers\Db::each($query) as $element) {
  
                  $this->setProgress($queue, $currentElement++ / $totalElements);
                  $match = Entry::find()->section('mapitOlfData')->ons($element->lsoacode)->one(); //one because we want one entry returned and lsoanames are unique
  
                  if ($match) {
                      // if we have a match then call the mapit api with the area id
                      $geoData = $this->fetchGeoJson($match->mapitAreaId);
  
                      $element->setFieldValue('mapitAreaId', $match->mapitAreaId);
                      $element->setFieldValue('geojson', $geoData);
  
                      if (!Craft::$app->getElements()->saveElement($element)) {
                          Craft::error(
                              ' || Couldnt save funding entry: ' . $element->id . ':' . $element->title . '| lsoa:' . $element->lsoacode,
                              __METHOD__
                          );
                      }else{
                          Craft::debug(
                              Craft::t(
                                  'mapitapi',
                                  'Sleeping for 2 secs ..saved funding entry: ' . $element->id . ':' . $element->title . '| lsoa:' . $element->lsoacode
                              ),
                              __METHOD__
                          );
                          sleep(1);
                      }
                  }
  
                  //}
              }
          } catch (Exception $e) {
              echo '[error]: '
                  . $e->getMessage()
                  . 'exception while processing '
                  . $currentElement . '/' . $totalElements
                  . ' - processing asset: ' . $element->title
                  . ' from field: ' . $element->lsoacode . PHP_EOL;
  
          }
      }
  
  ```
  
  **Tags:** plugin-development, queue
  
  > **Comments:**
  > **username_1**: The VPC might have more system memory, but your local one might have more assigned to PHP. What is your php.ini file's `memory_limit` setting set to locally? It looks like it's 256M on the VPC.  
  > **username_0**: Locally memory_limit is 512M, on VPS I set it to 512M and 1024M with the same result  
  > **username_2**: @username_0:  stupid question, did you edit the right config file? After you raised the memory limit, did the error message show the increased memory limit? Looks like you're running cpanel and easy apache? The CLI config is usually different than the web server one, not sure if this [doc](https://docs.cpanel.net/ea4/php/easyapache4-and-the-ea-php-cli-package/) is helpful... it's also possible your host disabled config editing...  
  > **username_0**: @username_2 fair question!. Yes and the chnage shows in craft php inspection panel. However I have better coding now which seems to have bypassed the issue. See https://craftcms.stackexchange.com/questions/38733/how-to-batch-a-large-queue-job/38734#38734  
  
  ---
  
  ## 0 answers
  
  