Answers.Body: <p>I think this is difficult to answer properly because it depends almost entirely on what the tasks and subtasks are actually doing.</p>
  
  <p>For example, the main task could do something that ate 400MB of memory and triggered 400 sub-tasks that did nothing but return null not using any memory.</p>
  
  <p>Alternatively, the main task could do nothing but trigger subtasks (using very little memory itself) but each subtask could use 400MB of memory.</p>
  
  <p>Obviously with subtasks you have to me more conscious of memory usage because usually there will be multiple subtasks and there is only ever one main task.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-03-15T23:45:10.497Z
Answers.Text: Thanks! That last paragraph is sort of what the wall of text was really aiming at. Which would be worse: A trillion _sub tasks_ (all belonging to the same main task) consuming 400MB each, or a trillion _main/regular/standalone tasks_ consuming 400MB each (with no sub tasks)?
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-03-15T23:41:27.670Z
Answers.LastActivityDate: 2016-03-15T23:41:27.670Z
Answers.UserDisplayName: Brad Bell
Body: <p>I'm currently implementing a task-based application, which pulls some data from an external API before creating a number of entries based on the payload.</p>
  
  <p>After creating the entries, the app will need to download a fair amount of images, storing the files as Assets and adding them to an Assets fields on one of the entries (each entry has a set of ~30 related images on an external server, that needs to be pulled).</p>
  
  <p>Currently, I'm running 1 task per entry, with a sub task for downloading all of the images related to each entry (1 step per image, so ~30 steps per sub task). The images are quite large – around 5MB each.</p>
  
  <p>The sub task takes an entry ID and an array of image URLs and metadata as its settings. Each step of the sub task looks like this:</p>
  
  <ol>
  <li>The entry the image will be related to is pulled from the entry ID.</li>
  <li>The task attempts to pull any existing Asset for the image (criteria based on filename)</li>
  <li>If Asset not found, the remote image is downloaded using Guzzle to a temporary folder, before the Asset is created using <code>craft()-&gt;assets-&gt;insertFileByLocalPath</code>.</li>
  <li>Another criteria to pull the newly created Asset.</li>
  <li>Asset is updated with the metadata (which could've changed, regardless of whether the Asset was previously saved)</li>
  <li>The IDs for any Assets in the entries Asset field is pulled using <code>$entry-&gt;assetField-&gt;ids()</code>.</li>
  <li>The new Asset's ID is added to the ID array, the asset field is updated with <code>$entry-&gt;getContent()-&gt;setAttribute('assetField', $ids);</code> and the entry is resaved through <code>craft()-&gt;entries-&gt;saveEntry</code>.</li>
  <li>The temporary image file is deleted using the IOHelper class.</li>
  </ol>
  
  <p>This is running on a 2GB droplet, where PHP/Craft has access to the full 2GB RAM – which I assumed should be plenty, though the server is running out of memory, intermittently. At one point, MySQL even crashed and had to be restarted :/ </p>
  
  <p>The memory issues never appear when Craft is running one of the regular, parent tasks – it always happens when one of the image sub tasks is running, which is why I'm debating my current setup.</p>
  
  <p>From Brad Bell's comment to <a href="https://craftcms.stackexchange.com/a/10356/1098">this answer</a>, I know that 1 task = 1 request, even if that task has multiple steps and/or sub tasks. What I take from that is that having loads of sub tasks is potentially more dangerous in terms of memory usage and possible timeout issues, than having loads of smaller regular tasks. <strong>Is this a correct assumption?</strong></p>
  
  <p>In closing, <em>which of the following approaches would be the safest</em>, in terms of memory usage (and/or other issues I might not have considered):</p>
  
  <ul>
  <li>One regular task per image (~30 tasks)</li>
  <li>One regular task per image set (~30 steps)</li>
  <li>One sub task per image (~30 sub tasks)</li>
  <li>One sub task per image set (~30 steps, my current implementation)</li>
  </ul>
  
  <p>I should mention that in this case, it doesn't really matter if the parent task completes before the image tasks run/complete, so I don't actually <em>need</em> to use sub tasks for the downloads.</p>
  
  <p>Also, <a href="https://craftcms.stackexchange.com/questions/7135/running-out-of-memory-in-my-task-based-plugin"><code>devMode</code> is currently <code>false</code> on the server</a> and I'm still having issues. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:20:21.130Z
Comments.Text: Just double checking, but you've also bumped this up as well? https://craftcms.com/docs/config-settings#phpMaxMemoryLimit That's an additional layer on top of whatever your php.ini file says.
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:23:02.997Z
Comments.Text: @BradBell Yeah. Currently set to `2048M`. I'm going to set up [Blackfire](https://blackfire.io) and hopefully figure out where the memory goes :)
Comments.UserDisplayName: Mats Mikkel Rummelhoff
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:26:05.847Z
Comments.Text: Yeah, hard to say. A lot of it is dependent on what your subtasks are actually doing. Blackfire is pretty slick for stuff like this.
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:28:08.623Z
Comments.Text: I'll add some detail on the sub tasks to my question – still curious about the basic issue, though; is it possible to say something general regarding resource consumption for lots of tasks vs. lots of sub tasks?
Comments.UserDisplayName: Mats Mikkel Rummelhoff
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:36:41.570Z
Comments.Text: I'll add an answer, but I don't think it's going to be very good. :)
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:38:14.310Z
Comments.Text: I have a feeling that the question itself kind of sucks, so no worries – thanks :D
Comments.UserDisplayName: Mats Mikkel Rummelhoff
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:48:05.343Z
Comments.Text: Probably worth mentioning you can affect some PHP garbage collection settings from php.ini that will affect memory usage, too. https://secure.php.net/manual/en/session.configuration.php
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-03-15T23:48:57.077Z
Comments.Text: That's a good tip, Brad – thanks. Will look into that, too.
Comments.UserDisplayName: Mats Mikkel Rummelhoff
Tags: <plugin-development><php><performance><tasks><memory>
Title: Memory usage in sub tasks vs. regular tasks
ThreadText: # Memory usage in sub tasks vs. regular tasks
  
  (Asked by: username_0 on 2016-03-15)
  
  I'm currently implementing a task-based application, which pulls some data from an external API before creating a number of entries based on the payload.
  
  After creating the entries, the app will need to download a fair amount of images, storing the files as Assets and adding them to an Assets fields on one of the entries (each entry has a set of ~30 related images on an external server, that needs to be pulled).
  
  Currently, I'm running 1 task per entry, with a sub task for downloading all of the images related to each entry (1 step per image, so ~30 steps per sub task). The images are quite large – around 5MB each.
  
  The sub task takes an entry ID and an array of image URLs and metadata as its settings. Each step of the sub task looks like this:
  
  1. The entry the image will be related to is pulled from the entry ID.
  2. The task attempts to pull any existing Asset for the image (criteria based on filename)
  3. If Asset not found, the remote image is downloaded using Guzzle to a temporary folder, before the Asset is created using `craft()->assets->insertFileByLocalPath`.
  4. Another criteria to pull the newly created Asset.
  5. Asset is updated with the metadata (which could've changed, regardless of whether the Asset was previously saved)
  6. The IDs for any Assets in the entries Asset field is pulled using `$entry->assetField->ids()`.
  7. The new Asset's ID is added to the ID array, the asset field is updated with `$entry->getContent()->setAttribute('assetField', $ids);` and the entry is resaved through `craft()->entries->saveEntry`.
  8. The temporary image file is deleted using the IOHelper class.
  
  This is running on a 2GB droplet, where PHP/Craft has access to the full 2GB RAM – which I assumed should be plenty, though the server is running out of memory, intermittently. At one point, MySQL even crashed and had to be restarted :/
  
  The memory issues never appear when Craft is running one of the regular, parent tasks – it always happens when one of the image sub tasks is running, which is why I'm debating my current setup.
  
  From username_1 username_1's comment to [this answer](https://craftcms.stackexchange.com/a/10356/1098), I know that 1 task = 1 request, even if that task has multiple steps and/or sub tasks. What I take from that is that having loads of sub tasks is potentially more dangerous in terms of memory usage and possible timeout issues, than having loads of smaller regular tasks. **Is this a correct assumption?**
  
  In closing, *which of the following approaches would be the safest*, in terms of memory usage (and/or other issues I might not have considered):
  
  * One regular task per image (~30 tasks)
  * One regular task per image set (~30 steps)
  * One sub task per image (~30 sub tasks)
  * One sub task per image set (~30 steps, my current implementation)
  
  I should mention that in this case, it doesn't really matter if the parent task completes before the image tasks run/complete, so I don't actually *need* to use sub tasks for the downloads.
  
  Also, [`devMode` is currently `false` on the server](https://craftcms.stackexchange.com/questions/7135/running-out-of-memory-in-my-task-based-plugin) and I'm still having issues.
  
  **Tags:** plugin-development, php, performance, tasks, memory
  
  > **Comments:**
  > **username_1**: Just double checking, but you've also bumped this up as well? https://craftcms.com/docs/config-settings#phpMaxMemoryLimit That's an additional layer on top of whatever your php.ini file says.  
  > **username_0**: @username_1 Yeah. Currently set to `2048M`. I'm going to set up [Blackfire](https://blackfire.io) and hopefully figure out where the memory goes :)  
  > **username_1**: Yeah, hard to say. A lot of it is dependent on what your subtasks are actually doing. Blackfire is pretty slick for stuff like this.  
  > **username_0**: I'll add some detail on the sub tasks to my question – still curious about the basic issue, though; is it possible to say something general regarding resource consumption for lots of tasks vs. lots of sub tasks?  
  > **username_1**: I'll add an answer, but I don't think it's going to be very good. :)  
  > **username_0**: I have a feeling that the question itself kind of sucks, so no worries – thanks :D  
  > **username_1**: Probably worth mentioning you can affect some PHP garbage collection settings from php.ini that will affect memory usage, too. https://secure.php.net/manual/en/session.configuration.php  
  > **username_0**: That's a good tip, username_1 – thanks. Will look into that, too.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-03-15. Score: 3 | Accepted answer
  
  I think this is difficult to answer properly because it depends almost entirely on what the tasks and subtasks are actually doing.
  
  For example, the main task could do something that ate 400MB of memory and triggered 400 sub-tasks that did nothing but return null not using any memory.
  
  Alternatively, the main task could do nothing but trigger subtasks (using very little memory itself) but each subtask could use 400MB of memory.
  
  Obviously with subtasks you have to me more conscious of memory usage because usually there will be multiple subtasks and there is only ever one main task.
  
  > **Comments:**
  > **username_0**: Thanks! That last paragraph is sort of what the wall of text was really aiming at. Which would be worse: A trillion _sub tasks_ (all belonging to the same main task) consuming 400MB each, or a trillion _main/regular/standalone tasks_ consuming 400MB each (with no sub tasks)?  
  
  ---
  
  