Answers.Body: <p>Remember that queue jobs are run in seperate processes from the code that enqueues them. So the 100 elements you're putting in the queue need to be serialized, saved to the database and then deserialized. I'm not sure if this will even work at all, but even if it does, it will cause memory issues as you have experienced.</p>
  <p>Instead of passing entire element objects to the queue, give it just an array of IDs:</p>
  <pre class="lang-php prettyprint-override"><code>$elementIds = Entry::find()
      -&gt;section('imdData')
      -&gt;orderBy('id')
      -&gt;ids();
  $queue = Craft::$app-&gt;getQueue();
  
  foreach (array_chunk($elementIds, 100) as $ids) {
      $job = $queue-&gt;push(new Importgeojson(['elementIds' =&gt; $ids]));
  }
  </code></pre>
  <p>Then you can use the IDs to get the corresponding elements from the database. Instead of querying for all elements at once (which will cause memory issues once again), process them one by one in your <code>Importgeojson</code> class:</p>
  <pre class="lang-php prettyprint-override"><code>$elementIds = $this-&gt;elementIds ?? [];
  foreach ($elementIds as $id) {
      $element = Entry::find()-&gt;anyStatus()-&gt;id($id)-&gt;one();
      if (!$element) continue;
      // process the current element …
  }
  </code></pre>
  <p>In general, when dealing with large amounts of entries/datasets, try to process them one by one to avoid memory issues.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-06T16:40:38.917Z
Answers.Text: Thanks @MoritzLost - gonna give this a whirl
Answers.UserDisplayName: joomkit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-06T23:02:42.517Z
Answers.Text: This is a fantastic improvement. It looks like it is moving through the queue nicely. Cant thank you enough for pointing the way. More testing to do now. THANK YOU
Answers.UserDisplayName: joomkit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-07T10:10:33.270Z
Answers.Text: @joomkit Glad this solution worked for you! If your issue is solved, would you mind marking my answer accepted? Thanks!
Answers.UserDisplayName: MoritzLost
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-06T15:45:43.967Z
Answers.LastActivityDate: 2021-12-06T15:45:43.967Z
Answers.UserDisplayName: MoritzLost
Body: <p>I am trying to batch a large queue job in my custom plugin.
  Here is my service that starts the queue.</p>
  <pre><code>    public function doBackGroundJob()
      {
  
          $query = Entry::find();
          if (!empty($this-&gt;criteria)) {
              Craft::configure($query, $this-&gt;criteria);
          }
          $query
              -&gt;section('imdData')
              -&gt;orderBy('id');
  
  
          foreach ($query-&gt;batch('100') as $elements) {
  
              $queue = Craft::$app-&gt;getQueue();
              $jobId = $queue-&gt;push(new Importgeojson(['elements' =&gt; $elements]));
              Craft::debug(
                  Craft::t(
                      'mapitapi',
                      'Started import geo json queue job id: {jobId}',
                      [
                          'jobId' =&gt; $jobId,
                      ]
                  ),
                  __METHOD__
              );
          }
      }
  </code></pre>
  <p>This calls a job importGeoJson</p>
  <pre><code>    public function execute($queue)
      {
  
          $totalElements = count($this-&gt;elements);
          $currentElement = 0;
  
  
          try {
              foreach ($this-&gt;elements as $element) {
              //foreach ($query-&gt;each() as $element) {
  
                  if(!$element-&gt;geojson){
  
  
                  $this-&gt;setProgress($queue, $currentElement++ / $totalElements);
                  // mapitOLF data contains mysocietiey mapit source where we can get the area id from the lsoa in order to get the geojson
                  $match = Entry::find()-&gt;section('mapitOlfData')-&gt;ons($element-&gt;lsoacode)-&gt;one(); //one because we want one entry returned and lsoanames are unique
  
                  if ($match) {
                      // if we have a match then call the mapit api with the area id
                      $geoData = $this-&gt;fetchGeoJson($match-&gt;mapitAreaId);
  
                      $element-&gt;setFieldValue('mapitAreaId', $match-&gt;mapitAreaId);
                      $element-&gt;setFieldValue('geojson', $geoData);
  
                      if (!Craft::$app-&gt;getElements()-&gt;saveElement($element)) {
                          Craft::error(
                              ' || Couldnt save funding entry: ' . $element-&gt;id . ':' . $element-&gt;title . '| lsoa:' . $element-&gt;lsoacode,
                              __METHOD__
                          );
                      }else{
                          Craft::debug(
                              Craft::t(
                                  'mapitapi',
                                  'Sleeping for 1 secs ..saved funding entry: ' . $element-&gt;id . ':' . $element-&gt;title . '| lsoa:' . $element-&gt;lsoacode
                              ),
                              __METHOD__
                          );
                          sleep(1);
                      }
                  }
  
                  }
              }
          } catch (Exception $e) {
              echo '[error]: '
                  . $e-&gt;getMessage()
                  . 'exception while processing '
                  . $currentElement . '/' . $totalElements
                  . ' - processing asset: ' . $element-&gt;title
                  . ' from field: ' . $element-&gt;lsoacode . PHP_EOL;
  
          }
      }
  </code></pre>
  <p>This just bombs out with memory issues straight away. PHP memory_limit is set to 512M</p>
  <p>Is there a better way to batch the job?
  Originally i was trying to load all the records at once and loop through them, relying on the async plugin to help with the tasks. This worked better than batching like this but still had memory issues.
  Any assistance appreciated :)</p>
  
Tags: <plugin-development><queue>
Title: How to batch a large queue job?
ThreadText: # How to batch a large queue job?
  
  (Asked by: username_0 on 2021-12-06)
  
  I am trying to batch a large queue job in my custom plugin.
  Here is my service that starts the queue.
  
  ```
      public function doBackGroundJob()
      {
  
          $query = Entry::find();
          if (!empty($this->criteria)) {
              Craft::configure($query, $this->criteria);
          }
          $query
              ->section('imdData')
              ->orderBy('id');
  
          foreach ($query->batch('100') as $elements) {
  
              $queue = Craft::$app->getQueue();
              $jobId = $queue->push(new Importgeojson(['elements' => $elements]));
              Craft::debug(
                  Craft::t(
                      'mapitapi',
                      'Started import geo json queue job id: {jobId}',
                      [
                          'jobId' => $jobId,
                      ]
                  ),
                  __METHOD__
              );
          }
      }
  
  ```
  
  This calls a job importGeoJson
  
  ```
      public function execute($queue)
      {
  
          $totalElements = count($this->elements);
          $currentElement = 0;
  
          try {
              foreach ($this->elements as $element) {
              //foreach ($query->each() as $element) {
  
                  if(!$element->geojson){
  
                  $this->setProgress($queue, $currentElement++ / $totalElements);
                  // mapitOLF data contains mysocietiey mapit source where we can get the area id from the lsoa in order to get the geojson
                  $match = Entry::find()->section('mapitOlfData')->ons($element->lsoacode)->one(); //one because we want one entry returned and lsoanames are unique
  
                  if ($match) {
                      // if we have a match then call the mapit api with the area id
                      $geoData = $this->fetchGeoJson($match->mapitAreaId);
  
                      $element->setFieldValue('mapitAreaId', $match->mapitAreaId);
                      $element->setFieldValue('geojson', $geoData);
  
                      if (!Craft::$app->getElements()->saveElement($element)) {
                          Craft::error(
                              ' || Couldnt save funding entry: ' . $element->id . ':' . $element->title . '| lsoa:' . $element->lsoacode,
                              __METHOD__
                          );
                      }else{
                          Craft::debug(
                              Craft::t(
                                  'mapitapi',
                                  'Sleeping for 1 secs ..saved funding entry: ' . $element->id . ':' . $element->title . '| lsoa:' . $element->lsoacode
                              ),
                              __METHOD__
                          );
                          sleep(1);
                      }
                  }
  
                  }
              }
          } catch (Exception $e) {
              echo '[error]: '
                  . $e->getMessage()
                  . 'exception while processing '
                  . $currentElement . '/' . $totalElements
                  . ' - processing asset: ' . $element->title
                  . ' from field: ' . $element->lsoacode . PHP_EOL;
  
          }
      }
  
  ```
  
  This just bombs out with memory issues straight away. PHP memory\_limit is set to 512M
  
  Is there a better way to batch the job?
  Originally i was trying to load all the records at once and loop through them, relying on the async plugin to help with the tasks. This worked better than batching like this but still had memory issues.
  Any assistance appreciated :)
  
  **Tags:** plugin-development, queue
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-06. Score: 3 | Accepted answer
  
  Remember that queue jobs are run in seperate processes from the code that enqueues them. So the 100 elements you're putting in the queue need to be serialized, saved to the database and then deserialized. I'm not sure if this will even work at all, but even if it does, it will cause memory issues as you have experienced.
  
  Instead of passing entire element objects to the queue, give it just an array of IDs:
  
  ```php
  $elementIds = Entry::find()
      ->section('imdData')
      ->orderBy('id')
      ->ids();
  $queue = Craft::$app->getQueue();
  
  foreach (array_chunk($elementIds, 100) as $ids) {
      $job = $queue->push(new Importgeojson(['elementIds' => $ids]));
  }
  
  ```
  
  Then you can use the IDs to get the corresponding elements from the database. Instead of querying for all elements at once (which will cause memory issues once again), process them one by one in your `Importgeojson` class:
  
  ```php
  $elementIds = $this->elementIds ?? [];
  foreach ($elementIds as $id) {
      $element = Entry::find()->anyStatus()->id($id)->one();
      if (!$element) continue;
      // process the current element …
  }
  
  ```
  
  In general, when dealing with large amounts of entries/datasets, try to process them one by one to avoid memory issues.
  
  > **Comments:**
  > **username_0**: Thanks @username_1 - gonna give this a whirl  
  > **username_0**: This is a fantastic improvement. It looks like it is moving through the queue nicely. Cant thank you enough for pointing the way. More testing to do now. THANK YOU  
  > **username_1**: @username_0 Glad this solution worked for you! If your issue is solved, would you mind marking my answer accepted? Thanks!  
  
  ---
  
  