Answers.Body: <p>After seeing this and hitting the same problem ourselves, we decided to create a plugin to index the assets from S3 in batches. We have not yet moved to using tasks in the plugin but it is on our roadmap. We know that Craft 3.x will have improvements to asset indexing but until then, please feel free to give our plugin a go if you are still experiencing issues with this. Our plugin can be downloaded from: <a href="https://github.com/a-digital/assetindexer" rel="nofollow">https://github.com/a-digital/assetindexer</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-31T16:07:47.540Z
Answers.LastActivityDate: 2016-10-31T16:07:47.540Z
Answers.UserDisplayName: Matt Shearing
Answers.Body: <p>Your best bet for a quick hacky solution would be to interrupt the indexing process, take a look at craft_assetindexdata table and then write a small plugin that calls craft()->assetindexing->processIndexForSource().</p>
  
  <p>You would get the sessionId from the craft_assetindexdata table and you'd be able to deduce how far the indexing has gotten by looking at the same table. All the entries that have recordId field set to a value have been processed already.</p>
  
  <p>This way you can batch it up however you want.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-03-01T19:43:05.997Z
Answers.LastActivityDate: 2015-03-01T19:43:05.997Z
Answers.UserDisplayName: Andris Sevcenko
Body: <p>I have 80,000+ files on S3 that need to be added to Craft. The Update Asset Indexes works, but takes over 24 hours and cannot be interrupted. Does anybody have a way to programmatically index assets like this in batches? </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-10T19:29:21.493Z
Comments.Text: Seems to be this sort of thing is best left to background [Tasks](http://buildwithcraft.com/classreference/models/TaskModel). Would love to find the time to write a plugin that did this, as we have a similar situation.
Comments.UserDisplayName: Chad W
Tags: <plugin-development><assets><php><index>
Title: Update Asset Index in Batches
ThreadText: # Update Asset Index in Batches
  
  (Asked by: username_0 on 2015-01-07)
  
  I have 80,000+ files on S3 that need to be added to Craft. The Update Asset Indexes works, but takes over 24 hours and cannot be interrupted. Does anybody have a way to programmatically index assets like this in batches?
  
  **Tags:** plugin-development, assets, php, index
  
  > **Comments:**
  > **username_1**: Seems to be this sort of thing is best left to background [Tasks](http://buildwithcraft.com/classreference/models/TaskModel). Would love to find the time to write a plugin that did this, as we have a similar situation.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2016-10-31. Score: 3
  
  After seeing this and hitting the same problem ourselves, we decided to create a plugin to index the assets from S3 in batches. We have not yet moved to using tasks in the plugin but it is on our roadmap. We know that Craft 3.x will have improvements to asset indexing but until then, please feel free to give our plugin a go if you are still experiencing issues with this. Our plugin can be downloaded from: <https://github.com/a-digital/assetindexer>
  
  
  ---
  
  ### Answer by username_3 on 2015-03-01. Score: 3
  
  Your best bet for a quick hacky solution would be to interrupt the indexing process, take a look at craft\_assetindexdata table and then write a small plugin that calls craft()->assetindexing->processIndexForSource().
  
  You would get the sessionId from the craft\_assetindexdata table and you'd be able to deduce how far the indexing has gotten by looking at the same table. All the entries that have recordId field set to a value have been processed already.
  
  This way you can batch it up however you want.
  
  
  ---
  
  