Answers.Body: <p>We ended up using this cli tool:
  <a href="https://github.com/rsanchez/craft-cli" rel="nofollow noreferrer">https://github.com/rsanchez/craft-cli</a></p>
  
  <p>We still had to tweak the php memory limit in the php config: <code>memory_limit 2048M</code></p>
  
  <p>But eventually we managed to index 21.000+ images this way. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-29T17:25:54.937Z
Answers.LastActivityDate: 2017-08-29T17:25:54.937Z
Answers.UserDisplayName: Youniteus
Body: <p>We uploaded a lot of images to the S3 bucket that is connected to our assets in Craft, but when we run "Update Asset Indexes" on the settings page, those images aren't added to the assets in Craft.</p>
  
  <p>I'm pretty sure that the settings for the asset source is correct (images that we upload via Craft are working). And it doesn't seem to be related to a time-out, because we updated the php and nginx settings, and now it finishes after ~70s instead of 60s.</p>
  
  <p>What are we doing wrong? 
  Is it even possible to import images this way?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-24T22:16:53.550Z
Comments.Text: Anything in your `craft/storage/runtime/logs` files when you update the asset indexes? If you have your browser's console open when you run it, do you get any non-200 O.K. responses?
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-25T19:40:24.653Z
Comments.Text: @BradBell Thanks for your comment. There was nothing in the craft logs, but the browser did returned a 502. I'm able to lower the timeout through nginx settings, but not increase it. So my current idea is that it's memory related. Maybe trying to import 21000+ images this way is to much. :o I'm thinking that we need to make a plugin to handle this amount of images.
Comments.UserDisplayName: Youniteus
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-25T19:50:37.580Z
Comments.Text: Just found this related question (https://craftcms.stackexchange.com/questions/6505/update-asset-index-in-batches) with a possible solution through a plugin (https://github.com/a-digital/assetindexer).
Comments.UserDisplayName: Youniteus
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-27T11:23:31.433Z
Comments.Text: Hmm that plugin isn't working to well. We're going to try to tweak with the craft code, and possibly create our own plugin.
Comments.UserDisplayName: Youniteus
Tags: <assets><php><s3><nginx><aws>
Title: Update Asset Indexes for new files in s3 not working
ThreadText: # Update Asset Indexes for new files in s3 not working
  
  (Asked by: username_0 on 2017-08-24)
  
  We uploaded a lot of images to the S3 bucket that is connected to our assets in Craft, but when we run "Update Asset Indexes" on the settings page, those images aren't added to the assets in Craft.
  
  I'm pretty sure that the settings for the asset source is correct (images that we upload via Craft are working). And it doesn't seem to be related to a time-out, because we updated the php and nginx settings, and now it finishes after ~70s instead of 60s.
  
  What are we doing wrong?
  Is it even possible to import images this way?
  
  **Tags:** assets, php, s3, nginx, aws
  
  > **Comments:**
  > **username_1**: Anything in your `craft/storage/runtime/logs` files when you update the asset indexes? If you have your browser's console open when you run it, do you get any non-200 O.K. responses?  
  > **username_0**: @username_1 Thanks for your comment. There was nothing in the craft logs, but the browser did returned a 502. I'm able to lower the timeout through nginx settings, but not increase it. So my current idea is that it's memory related. Maybe trying to import 21000+ images this way is to much. :o I'm thinking that we need to make a plugin to handle this amount of images.  
  > **username_0**: Just found this related question (https://craftcms.stackexchange.com/questions/6505/update-asset-index-in-batches) with a possible solution through a plugin (https://github.com/a-digital/assetindexer).  
  > **username_0**: Hmm that plugin isn't working to well. We're going to try to tweak with the craft code, and possibly create our own plugin.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2017-08-29. Score: 0 | Accepted answer
  
  We ended up using this cli tool:
  <https://github.com/rsanchez/craft-cli>
  
  We still had to tweak the php memory limit in the php config: `memory_limit 2048M`
  
  But eventually we managed to index 21.000+ images this way.
  
  
  ---
  
  