Answers.Body: <p>I've recently looked at this problem myself due to having feeds that lack any pagination capability e.g. CSV file and while you can offload the queue to a completely separate worker process to avoid issues with interfering with the web front end delivery for long running tasks, having a long running task for lots of data isn't performant, if it fails you are restarting from the beginning and will likely take much longer than say dividing 1000 items into 10 chunks smaller chunks due to processing overhead.</p>
  <p>As the original answer mentioned, chunking is one of the first steps. Because FeedMe in Craft 3 and 4 support <code>limit</code> and <code>offset</code> options you can in fact leverage the existing queue job, by passing in your desired chunking options, without having to reinvent the wheel, you would need a custom console command/service to leverage this however.</p>
  <p>My approach with feeds that don't support pagination is to essentially:</p>
  <ol>
  <li>Create a console command that has a chunk parameter e.g. 100 per chunk, could be customisable for different feeds if required.</li>
  <li>Get the feed and call getFeedMeData() to get all the feed data so we can work with it in a loop.</li>
  <li>Get the total items in the feed, so we know the overall items we have for limit/offset logic to come later.</li>
  <li>Set the limit to the chunk size encountered in each loop. Something like <code>count($chunk)</code></li>
  <li>Set the offset to a calculation of <code>$index * $chunkSize</code>. Assuming you had a 100 chunk size, for the first chunk the offset is 0, the next would be 100, 200 etc. For the last chunk, you might want to change the offset calculation to be <code>$totalItems - $chunkSize</code>, so it doesn't ever exceed the actual total items, as the last chunk may not be necessarily align to the defined maximum chunk size equally, say if you had 1501 items, you'd have 15 queue jobs with 100 items, with then one queue job left with just 1.</li>
  </ol>
  <p>You can then create create a FeedMe queue job with a loop similar to this:</p>
  <pre class="lang-php prettyprint-override"><code>$feed = FeedMe::$plugin-&gt;feeds-&gt;getFeedById($feedId);
  $feedData = $feed-&gt;getFeedData();
  $totalItems = count($feedData);
  $chunkedFeedData = array_chunk($feedData, $this-&gt;chunkSize);
  $steps = count($chunkedFeedData);
  
  foreach ($chunkedFeedData as $index =&gt; $chunk) {
      $step = $index + 1;
      $chunkCount = count($chunk);
      $offset = $index * $chunkCount;
  
      Queue::push(new FeedImport([
          'feed' =&gt; $feed,
          'limit' =&gt; $chunkCount,
          'offset' =&gt; $offset,
          'continueOnError' =&gt; $this-&gt;continueOnError,
          'description' =&gt; &quot;Running $feed-&gt;name step $step of $steps&quot;
       ]));
  }
  </code></pre>
  <p>This is just an example, you will want to add some checking on $feed being valid and $feedData having data.</p>
  <p>We can also override the description so each individual queue job is labelled accordingly, so it's easier to keep track and know which part of the sequence the queue is running, given we are dealing with many queue jobs with chunking.</p>
  <p>There is a potential downside to this method, getFeedMeData() will be getting called many times. As the default FeedMe queue job calls getFeedMeData in it's queue job logic, so you are potentially calling the feed many times than you need to when using limit and offset values. Alternatively, you could implement some form of data caching, so you cache the feed data response for a period of time before you chunk it, however for a large amount of records, caching that amount of data could be an issue.</p>
  <p>Alternatively, you could write a modified FeedMe import queue job that doesn't call getFeedMeData() in the execute part and instead the data is simply passed to it, given you've already had to call getFeedMeData() once for the chunking logic to know how many chunks you have, the chunk size to break each smaller queue job down by etc.</p>
  <p>Either way approaching a large data feed into smaller queue jobs is the way to go. The general rule for performance is splitting down a large amount of items into smaller batches is going to be faster and less memory intensive than doing it without chunking. Chunking will be much kinder to memory usage per queue job and in the event a queue job fails, only a batch of x amount of steps has failed not the whole lot. Equally with ttr and retry you can handle this, even FeedMe itself supports this.</p>
  <p>The other consideration would be around the time a chunked Feed Me queue job would take. Even if it's chunked it's still probably a long time for 1 million records, therefore you may want to dedicate a queue worker specifically for feed processing and run another queue for general Craft queue jobs, essentially two workers running two different queues. You are very likely to create a massive backlog of general Craft queue jobs queuing up while your large feed import is running. So multiple workers/queues are going to help here too.</p>
  <p>Both Craft CMS (since 3.4) and FeedMe support running under a completely different custom queue, you can find the information around this here: <a href="https://github.com/craftcms/cms/issues/5492" rel="nofollow noreferrer">https://github.com/craftcms/cms/issues/5492</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-03T11:57:46.457Z
Answers.LastActivityDate: 2022-06-22T21:20:55.300Z
Answers.LastEditDate: 2022-06-22T21:20:55.300Z
Answers.UserDisplayName: James White
Answers.Body: <p>You could write a service method that reads the import file and divides it into chunks, then it could dispatch a queue job for each chunk which does the actual entry creation/update. This way you could scale the resources and amount of queue workers available to work through the queue and even Play with the chunk size.</p>
  <p>In the queue job you might want to implement proper exception handling to keep track of unsuccessful entry operations to handle them somehow.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-23T10:17:49.347Z
Answers.Text: In addition to this good advice, if as you say a large percentage will be duplicates (I assume this refers to products that already exist in Craft but have not been altered in the import source and therefore do not require re-importing), I'd recommend storing a hash/checksum in a custom field against each product so that you know whether you can skip the processing on the next import - this will save you lots of processing power and time taken to complete the import. I do a monthly import of around 60k products into Craft Commerce and this approach saves a significant amount of time.
Answers.UserDisplayName: James Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-31T21:05:24.087Z
Answers.Text: @Johannes thank you for your suggestion. Would that be done as a custom module/plugin in Craft? Do you know of any documentation/tutorials that might help me do this?
Answers.UserDisplayName: fxfuture
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-31T21:17:43.840Z
Answers.Text: @fxfuture There is a whole area for extending Craft in the docs: https://craftcms.com/docs/3.x/extend/#modules-vs-plugins . Also, you should have a look at https://pluginfactory.io/ to generate a module scaffolding. You may structure your code in a Service class that utilizes an import Task. The service could be triggered via a Controller (HTTP) or a Console command. Regarding the hashing, you may have a look at Yii's concept of behaviors.
Answers.UserDisplayName: Johannes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-31T21:19:51.777Z
Answers.Text: If you want to trigger the import Service on a regular basis I would suggest creating a thin Console command. This way, you could create a Cron job that runs the command whenever you want.
Answers.UserDisplayName: Johannes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-31T22:03:57.670Z
Answers.Text: @Johannes that's really helpful thanks. I will take a look. And do you think the performance will be able to handle this many entries efficiently?
Answers.UserDisplayName: fxfuture
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-22T13:01:26.340Z
Answers.LastActivityDate: 2021-05-22T13:01:26.340Z
Answers.UserDisplayName: Johannes
Body: <p>I have a shopping product feed that is generated every day with around a million products. This can be in any format needed - CSV, JSON, XML etc, and will live on an S3 bucket.</p>
  <p>Each day, I need to import this feed to entries in Craft and assign them categories etc. A large percentage of the products will be duplicates but it may need to update data within the product.</p>
  <p>What would be the best way to import this large of a data set?</p>
  <p>I have used FeedMe in the past, but I'm not sure whether it could handle this many products.</p>
  <p>Thanks in advance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-28T11:01:36.560Z
Comments.Text: Hey @fxfuture, do you have any further questions how to implement my suggestion?
Comments.UserDisplayName: Johannes
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-31T21:06:13.827Z
Comments.Text: @Johannes thank you for your suggestion. Would that be done as a custom module/plugin in Craft? Do you know of any documentation/tutorials that might help me do this?
Comments.UserDisplayName: fxfuture
Tags: <craft3><plugin-feedme><import>
Title: Import large number of entries every day - 1 million
ThreadText: # Import large number of entries every day - 1 million
  
  (Asked by: username_0 on 2021-05-20)
  
  I have a shopping product feed that is generated every day with around a million products. This can be in any format needed - CSV, JSON, XML etc, and will live on an S3 bucket.
  
  Each day, I need to import this feed to entries in Craft and assign them categories etc. A large percentage of the products will be duplicates but it may need to update data within the product.
  
  What would be the best way to import this large of a data set?
  
  I have used FeedMe in the past, but I'm not sure whether it could handle this many products.
  
  Thanks in advance.
  
  **Tags:** craft3, plugin-feedme, import
  
  > **Comments:**
  > **username_1**: Hey @username_0, do you have any further questions how to implement my suggestion?  
  > **username_0**: @username_1 thank you for your suggestion. Would that be done as a custom module/plugin in Craft? Do you know of any documentation/tutorials that might help me do this?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-05-22. Score: 4 | Accepted answer
  
  You could write a service method that reads the import file and divides it into chunks, then it could dispatch a queue job for each chunk which does the actual entry creation/update. This way you could scale the resources and amount of queue workers available to work through the queue and even Play with the chunk size.
  
  In the queue job you might want to implement proper exception handling to keep track of unsuccessful entry operations to handle them somehow.
  
  > **Comments:**
  > **username_3**: In addition to this good advice, if as you say a large percentage will be duplicates (I assume this refers to products that already exist in Craft but have not been altered in the import source and therefore do not require re-importing), I'd recommend storing a hash/checksum in a custom field against each product so that you know whether you can skip the processing on the next import - this will save you lots of processing power and time taken to complete the import. I do a monthly import of around 60k products into Craft Commerce and this approach saves a significant amount of time.  
  > **username_0**: @username_1 thank you for your suggestion. Would that be done as a custom module/plugin in Craft? Do you know of any documentation/tutorials that might help me do this?  
  > **username_1**: @username_0 There is a whole area for extending Craft in the docs: https://craftcms.com/docs/3.x/extend/#modules-vs-plugins . Also, you should have a look at https://pluginfactory.io/ to generate a module scaffolding. You may structure your code in a Service class that utilizes an import Task. The service could be triggered via a Controller (HTTP) or a Console command. Regarding the hashing, you may have a look at Yii's concept of behaviors.  
  > **username_1**: If you want to trigger the import Service on a regular basis I would suggest creating a thin Console command. This way, you could create a Cron job that runs the command whenever you want.  
  > **username_0**: @username_1 that's really helpful thanks. I will take a look. And do you think the performance will be able to handle this many entries efficiently?  
  
  ---
  
  ### Answer by username_2 on 2022-06-03. Score: 1
  
  I've recently looked at this problem myself due to having feeds that lack any pagination capability e.g. CSV file and while you can offload the queue to a completely separate worker process to avoid issues with interfering with the web front end delivery for long running tasks, having a long running task for lots of data isn't performant, if it fails you are restarting from the beginning and will likely take much longer than say dividing 1000 items into 10 chunks smaller chunks due to processing overhead.
  
  As the original answer mentioned, chunking is one of the first steps. Because FeedMe in Craft 3 and 4 support `limit` and `offset` options you can in fact leverage the existing queue job, by passing in your desired chunking options, without having to reinvent the wheel, you would need a custom console command/service to leverage this however.
  
  My approach with feeds that don't support pagination is to essentially:
  
  1. Create a console command that has a chunk parameter e.g. 100 per chunk, could be customisable for different feeds if required.
  2. Get the feed and call getFeedMeData() to get all the feed data so we can work with it in a loop.
  3. Get the total items in the feed, so we know the overall items we have for limit/offset logic to come later.
  4. Set the limit to the chunk size encountered in each loop. Something like `count($chunk)`
  5. Set the offset to a calculation of `$index * $chunkSize`. Assuming you had a 100 chunk size, for the first chunk the offset is 0, the next would be 100, 200 etc. For the last chunk, you might want to change the offset calculation to be `$totalItems - $chunkSize`, so it doesn't ever exceed the actual total items, as the last chunk may not be necessarily align to the defined maximum chunk size equally, say if you had 1501 items, you'd have 15 queue jobs with 100 items, with then one queue job left with just 1.
  
  You can then create create a FeedMe queue job with a loop similar to this:
  
  ```php
  $feed = FeedMe::$plugin->feeds->getFeedById($feedId);
  $feedData = $feed->getFeedData();
  $totalItems = count($feedData);
  $chunkedFeedData = array_chunk($feedData, $this->chunkSize);
  $steps = count($chunkedFeedData);
  
  foreach ($chunkedFeedData as $index => $chunk) {
      $step = $index + 1;
      $chunkCount = count($chunk);
      $offset = $index * $chunkCount;
  
      Queue::push(new FeedImport([
          'feed' => $feed,
          'limit' => $chunkCount,
          'offset' => $offset,
          'continueOnError' => $this->continueOnError,
          'description' => "Running $feed->name step $step of $steps"
       ]));
  }
  
  ```
  
  This is just an example, you will want to add some checking on $feed being valid and $feedData having data.
  
  We can also override the description so each individual queue job is labelled accordingly, so it's easier to keep track and know which part of the sequence the queue is running, given we are dealing with many queue jobs with chunking.
  
  There is a potential downside to this method, getFeedMeData() will be getting called many times. As the default FeedMe queue job calls getFeedMeData in it's queue job logic, so you are potentially calling the feed many times than you need to when using limit and offset values. Alternatively, you could implement some form of data caching, so you cache the feed data response for a period of time before you chunk it, however for a large amount of records, caching that amount of data could be an issue.
  
  Alternatively, you could write a modified FeedMe import queue job that doesn't call getFeedMeData() in the execute part and instead the data is simply passed to it, given you've already had to call getFeedMeData() once for the chunking logic to know how many chunks you have, the chunk size to break each smaller queue job down by etc.
  
  Either way approaching a large data feed into smaller queue jobs is the way to go. The general rule for performance is splitting down a large amount of items into smaller batches is going to be faster and less memory intensive than doing it without chunking. Chunking will be much kinder to memory usage per queue job and in the event a queue job fails, only a batch of x amount of steps has failed not the whole lot. Equally with ttr and retry you can handle this, even FeedMe itself supports this.
  
  The other consideration would be around the time a chunked Feed Me queue job would take. Even if it's chunked it's still probably a long time for 1 million records, therefore you may want to dedicate a queue worker specifically for feed processing and run another queue for general Craft queue jobs, essentially two workers running two different queues. You are very likely to create a massive backlog of general Craft queue jobs queuing up while your large feed import is running. So multiple workers/queues are going to help here too.
  
  Both Craft CMS (since 3.4) and FeedMe support running under a completely different custom queue, you can find the information around this here: <https://github.com/craftcms/cms/issues/5492>
  
  
  ---
  
  