Answers.Body: <p>I would remove the overhead of the CMS all together by using an app like Filezilla, Cyberduck or similar app to move files. There are plenty of tutorials out there on how to connect to an S3 bucket using one of these apps (eg. <a href="https://stackoverflow.com/questions/45293159/how-to-setup-filezilla-pro-to-access-to-my-s3-bucket">https://stackoverflow.com/questions/45293159/how-to-setup-filezilla-pro-to-access-to-my-s3-bucket</a>)</p>
  
  <p>After you do this you will most likely need to update your asset indexes in the control panel:</p>
  
  <p>Utilities → Asset Indexes (or Settings → Update Asset Indexes if you're on Craft 2)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-04T22:32:18.843Z
Answers.Text: I am using Transmit to transfer the actual files and then updating the index of the files in the Holding folder linked in Craft. The problem is when I move the files from Holding to the permanent folder. I have the Holding folder so I can index only 2,000 at a time vs the permanent folder which grows to over 26,000 by years end.
Answers.UserDisplayName: Lori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-08T16:50:45.037Z
Answers.Text: I've been doing some research on Assets, and I thought this article (https://www.vaersaagod.no/en/9-tips-for-speeding-up-your-imager-transforms-in-craftcms) by Andre Elvan may be helpful in understanding assets and image transforms in relationship to the server. It may or may not answer the question you posed here, but it's a good overall reference on the subject.
Answers.UserDisplayName: Vaughn D. Taylor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-09T16:13:47.273Z
Answers.Text: Thanks @vaughn-d-taylor .  I read through the document you referenced and looked into the setting referenced but it seems like I'm ok there.  My `max_execution_timeout` is 90 and my `max_requests` look high enough too.  My `memory_limit` is 256 so that looks good.  It takes about 10 seconds to move 1 file from Holding to Live if the asset folder located on my server.  It's taking 20 seconds to move 1 file from Holding to Live if the asset folder is located on S3.  Could it be that my search-index is so large that it's taking a while to update it with the new asset path?
Answers.UserDisplayName: Lori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-09T18:29:27.950Z
Answers.Text: I cannot say for sure if it's your search index causing the issue. However, you may try increasing your max_execution_time = 300, max_input_time = 600, and memory_limit = 4G (or more if you have it) just as an experiment. I've run into issues with a very large DB and CraftCMS. The only way I could deal with the issue was to crank up those settings.
Answers.UserDisplayName: Vaughn D. Taylor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-09T22:50:34.240Z
Answers.Text: My server company updated the items as @vaughn-d-taylor suggested.  I tested it out with one file and the speed to transfer took 50% less time the first time but then the second time it took close to 2 minutes.  My phperrors.log file didn't show any time out errors.  But my queue.log file showed a lot of `craft_queue` and `craft_templatecaches` and `craft_templatecachequeries`.  The `craft_queue` has `delay` as part of the query.
Answers.UserDisplayName: Lori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T15:48:03.157Z
Answers.Text: I looked in my DB tables and those tables are empty.  Would a queue plugin help?  I found AsyncQueue and Highfire in the Craft 3 plugins area.
Answers.UserDisplayName: Lori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-04T22:29:28.463Z
Answers.LastActivityDate: 2019-01-04T22:29:28.463Z
Answers.UserDisplayName: Vaughn D. Taylor
Body: <p>In Craft 3, I keep running into an issue when I try to move a large amount of files (around 200) by dragging and dropping from my “Holding” folder into a “Live” folder using Craft Assets section. Both folders are hosted on Amazon S3.  Even moving three files seems to take a really long time (about a minute).</p>
  
  <p>After checking the server logs it looks like I am filling my max apache connection due to the search-index-perform-action.  The error suggested I increase my “MaxRequestWorkers” setting.</p>
  
  <p>I move about 2,000 files every month and never had a problem on Craft 2.  I don’t know if should change the “MaxRequestWorkers” setting or if there is something else I should consider doing.  If I should change the setting, where is it, I can't seem to find it.</p>
  
  <p>I use Transmit FTP to initially move the files into the Holding folder and then index the assets in that folder. Then in Craft CMS control panel, I drag the files from Holding folder to the main Live folder so that the asset index gets updated.</p>
  
  <hr>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I looked in my <code>phperror.log</code> file and found these three types of errors occuring:</p>
  
  <pre><code>PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1048576 bytes) in /storage/avxxxxx/www/public_html/vendor/craftcms/cms/src/helpers/DateTimeHelper.php on line 231
  
  PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /storage/avxxxxx/www/public_html/vendor/craftcms/cms/src/elements/db/ElementQuery.php on line 2215
  
  PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem/src/Util.php on line 322
  
  PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php on line 294
  
  PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem/src/Util/ContentListingFormatter.php on line 52
  </code></pre>
  
  <p>It seems like when I move a file from Holding (950 files) to Live (20,000) files is where things get hung up.</p>
  
  <ul>
  <li>Holding -> Live = 1 file takes 7 seconds to move</li>
  <li>Live -> Holding = 1 file takes 20 seconds to move</li>
  </ul>
  
  <p>Could it be an issue with the cache or search-index?</p>
  
  <p>Here are my current settings if it helps:</p>
  
  <pre><code>memory_limit 256M
  
  max_execution_time = 90
  
  Max Requests = Per Child: 300 - Keep Alive: on - Max Per Connection: 150
  
  Timeouts = Connection: 10 - Keep-Alive: 2
  </code></pre>
  
Tags: <assets><error-message><apache>
Title: Craft 3 - Moving assets between folders causes apache connection to max out
ThreadText: # Craft 3 - Moving assets between folders causes apache connection to max out
  
  (Asked by: username_0 on 2019-01-04)
  
  In Craft 3, I keep running into an issue when I try to move a large amount of files (around 200) by dragging and dropping from my “Holding” folder into a “Live” folder using Craft Assets section. Both folders are hosted on Amazon S3. Even moving three files seems to take a really long time (about a minute).
  
  After checking the server logs it looks like I am filling my max apache connection due to the search-index-perform-action. The error suggested I increase my “MaxRequestWorkers” setting.
  
  I move about 2,000 files every month and never had a problem on Craft 2. I don’t know if should change the “MaxRequestWorkers” setting or if there is something else I should consider doing. If I should change the setting, where is it, I can't seem to find it.
  
  I use Transmit FTP to initially move the files into the Holding folder and then index the assets in that folder. Then in Craft CMS control panel, I drag the files from Holding folder to the main Live folder so that the asset index gets updated.
  
  ---
  
  **UPDATE**
  
  I looked in my `phperror.log` file and found these three types of errors occuring:
  
  ```
  PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1048576 bytes) in /storage/avxxxxx/www/public_html/vendor/craftcms/cms/src/helpers/DateTimeHelper.php on line 231
  
  PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /storage/avxxxxx/www/public_html/vendor/craftcms/cms/src/elements/db/ElementQuery.php on line 2215
  
  PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem/src/Util.php on line 322
  
  PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php on line 294
  
  PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem/src/Util/ContentListingFormatter.php on line 52
  
  ```
  
  It seems like when I move a file from Holding (950 files) to Live (20,000) files is where things get hung up.
  
  * Holding -> Live = 1 file takes 7 seconds to move
  * Live -> Holding = 1 file takes 20 seconds to move
  
  Could it be an issue with the cache or search-index?
  
  Here are my current settings if it helps:
  
  ```
  memory_limit 256M
  
  max_execution_time = 90
  
  Max Requests = Per Child: 300 - Keep Alive: on - Max Per Connection: 150
  
  Timeouts = Connection: 10 - Keep-Alive: 2
  
  ```
  
  **Tags:** assets, error-message, apache
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-04. Score: 0
  
  I would remove the overhead of the CMS all together by using an app like Filezilla, Cyberduck or similar app to move files. There are plenty of tutorials out there on how to connect to an S3 bucket using one of these apps (eg. <https://stackoverflow.com/questions/45293159/how-to-setup-filezilla-pro-to-access-to-my-s3-bucket>)
  
  After you do this you will most likely need to update your asset indexes in the control panel:
  
  Utilities → Asset Indexes (or Settings → Update Asset Indexes if you're on Craft 2)
  
  > **Comments:**
  > **username_0**: I am using Transmit to transfer the actual files and then updating the index of the files in the Holding folder linked in Craft. The problem is when I move the files from Holding to the permanent folder. I have the Holding folder so I can index only 2,000 at a time vs the permanent folder which grows to over 26,000 by years end.  
  > **username_1**: I've been doing some research on Assets, and I thought this article (https://www.vaersaagod.no/en/9-tips-for-speeding-up-your-imager-transforms-in-craftcms) by Andre Elvan may be helpful in understanding assets and image transforms in relationship to the server. It may or may not answer the question you posed here, but it's a good overall reference on the subject.  
  > **username_0**: Thanks @username_1-d-username_1 .  I read through the document you referenced and looked into the setting referenced but it seems like I'm ok there.  My `max_execution_timeout` is 90 and my `max_requests` look high enough too.  My `memory_limit` is 256 so that looks good.  It takes about 10 seconds to move 1 file from Holding to Live if the asset folder located on my server.  It's taking 20 seconds to move 1 file from Holding to Live if the asset folder is located on S3.  Could it be that my search-index is so large that it's taking a while to update it with the new asset path?  
  > **username_1**: I cannot say for sure if it's your search index causing the issue. However, you may try increasing your max_execution_time = 300, max_input_time = 600, and memory_limit = 4G (or more if you have it) just as an experiment. I've run into issues with a very large DB and CraftCMS. The only way I could deal with the issue was to crank up those settings.  
  > **username_0**: My server company updated the items as @username_1-d-username_1 suggested.  I tested it out with one file and the speed to transfer took 50% less time the first time but then the second time it took close to 2 minutes.  My phperrors.log file didn't show any time out errors.  But my queue.log file showed a lot of `craft_queue` and `craft_templatecaches` and `craft_templatecachequeries`.  The `craft_queue` has `delay` as part of the query.  
  > **username_0**: I looked in my DB tables and those tables are empty.  Would a queue plugin help?  I found AsyncQueue and Highfire in the Craft 3 plugins area.  
  
  ---
  
  