Answers.Body: <p>OK, finally answering my own question here as it seems to not be an issue with the latest Craft 2.5 builds when <code>devMode</code> is turned off, but is still an issue when <code>devMode</code> is on.</p>
  
  <p>As far as I can tell its something to do with the Craft logging tool spiralling out of control, but thats as far as I got.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-15T11:38:08.443Z
Answers.Text: Just to say was having similar issues until disabling dev mode
Answers.UserDisplayName: Cole Henley
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-15T11:39:09.707Z
Answers.Text: Yeah took me ages to work out!
Answers.UserDisplayName: Josh Angell
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-03T11:31:52.630Z
Answers.Text: https://craftcms.com/support/dev-mode This should cast some light on dev mode. Explains what is being logged and more
Answers.UserDisplayName: Krzysztof Boduch
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-28T19:03:15.387Z
Answers.Text: Also want to echo these guys, disabling dev mode fixed my memory issue with my tasks plugin. @JoshAngell have you contacted P&T about this?
Answers.UserDisplayName: Chase Giunta
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-28T19:14:25.160Z
Answers.Text: @ChaseGiunta yeah I have an open ticket with Brad, I think as its not likely to creep into production (or shouldn't rather) and its a nightmare to test against it's not going to be sorted any time soon. I had to profile it to find it.
Answers.UserDisplayName: Josh Angell
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-15T11:28:36.493Z
Answers.LastActivityDate: 2015-12-15T11:28:36.493Z
Answers.UserDisplayName: Josh Angell
Body: <p>I am writing a plugin that imports a large amount of data form an API into some custom element types. I have it all set up and working nicely with Tasks however I am running into a fairly big issue with memory.</p>
  
  <p>The plugin runs in the following way:</p>
  
  <ol>
  <li>Controlling Task gets set off.</li>
  <li>It sets off a SubTask twice that gets two different sets of data from the API and saves it to disk.</li>
  <li>It then sets off another bunch of SubTasks that import that data into my custom element types one element at a time - the number of Tasks at this point depends on how large the saved API data is as I split it into chunks of 100.</li>
  </ol>
  
  <p>In my testing I am skipping point 2 from above, so it is only reading stuff from disk, importing it and repeating. I am finding that no matter what I do I can't stop the memory usage building up. I have used <code>unset()</code> in various places which has helped a little bit and have turned off the <code>cacheElementQueries</code> config parameter using <code>craft()-&gt;config-&gt;set('cacheElementQueries', false);</code> which has also helped but I'm still running into problems.</p>
  
  <p><strong>Some numbers</strong></p>
  
  <ul>
  <li>Resulting files from the API are 2.5MB and 5.3MB of json.</li>
  <li>Total elements to import from each are 1200 and 2000 respectively.</li>
  <li>As mentioned above I'm splitting the sets of data in to chunks of 100 to try and conserve memory.</li>
  <li>php.ini <code>memory_limit</code> setting is 512M</li>
  </ul>
  
  <p><strong>Conclusion</strong></p>
  
  <p>I feel like I'm missing something obvious as php doesn't dump the memory until all the Tasks have run, or at least that is what it seems like. I kind of thought that after each Task or SubTask had completed the memory for that would be dumped.</p>
  
  <p>I have to confess I'm still a bit new to serious programming and am only using <code>htop</code> to monitor the process and can see it going up continually until it reaches my php.ini <code>memory_limit</code> setting. I know I should probably use xdebug or something but that kind of scares me...</p>
  
  <p>I've also read this: <a href="https://craftcms.stackexchange.com/questions/740/memory-issue-when-updating-all-entries-from-a-channel-allowed-memory-size-of-x">Memory issue when updating all entries from a channel (allowed memory size of X bytes exhausted (tried to allocate Y bytes))</a></p>
  
  <p>If anyone has any ideas however wacky to help with plugging my leaky ship I'd be most grateful! I can post up code samples if it will help but I didn't want to make this post too lengthy.</p>
  
  <hr>
  
  <h2>Update</h2>
  
  <p>Here is a cut-down set of functions that are used in the process, I've not included lots of little bits that are either just doing processing on non-large variables or other mundane tasks.</p>
  
  
  
  <p><strong>Main Task</strong></p>
  
  <pre class="lang-php prettyprint-override"><code>public function runStep($step)
  {
  
      // Sync the stored API data to the Craft elements
      return $this-&gt;runSubTask('MyPlugin_Sync', null, array(
          'apiElement'         =&gt; $this-&gt;_apiElements[$step]['apiElement'],
          'apiElementSettings' =&gt; $this-&gt;_apiElements[$step]['apiElementSettings'],
          'apiService'         =&gt; $service,
          'offset'             =&gt; $this-&gt;_apiElements[$step]['offset']
      ));
  
  }
  </code></pre>
  
  <p><strong>Sync Task</strong></p>
  
  <pre class="lang-php prettyprint-override"><code>public function getTotalSteps()
  {
  
      $this-&gt;_apiData = craft()-&gt;myPlugin_ApiService-&gt;getElementsFromFile($this-&gt;_apiElement, $this-&gt;_offset, 100);
  
      // Check we got something
      if ( is_array($this-&gt;_apiData) )
      {
          return count($this-&gt;_apiData);
      }
      else
      {
          return 0;
      }
  
  }
  
  public function runStep($step)
  {
  
      return craft()-&gt;myPlugin_ImportService-&gt;importElement($this-&gt;_apiService, $this-&gt;_apiElementSettings, array_shift($this-&gt;_apiData));
  
  }
  </code></pre>
  
  <p><strong>API Service</strong></p>
  
  <pre class="lang-php prettyprint-override"><code>public function getElementsFromFile($apiElement, $offset = 0, $limit = null)
  {
  
      // Get the file
      $filename = $this-&gt;_storage . $apiElement . 's.json';
  
      $file = IOHelper::getFile($filename);
  
      $data = json_decode($file-&gt;getContents(), true);
  
      // Try and help the memory
      unset($file);
  
      // Now select the right portion from it using our offset amount
      return array_slice($data, $offset, $limit);
  
  }
  
  public function mapData($data, $settings)
  {
  
      $return = array();
  
      // Boring mapping code using $data
  
      unset($data);
  
      return $return;
  
  }
  </code></pre>
  
  <p><strong>Import Service</strong></p>
  
  <pre class="lang-php prettyprint-override"><code>public function importElement($apiService, $apiElementSettings, $apiData)
  {
  
      // using the API service, map the data to our fields/attributes held in the plugin settings
      $mappedData = craft()-&gt;myPlugin_ApiService-&gt;mapData($apiData, $apiElementSettings);
  
      // We’re done with this now, save our memory
      unset($apiData);
  
      // Standard code for getting an existing element and updating it
      // or creating a new one if none exist
  
      // Save the thing
      return craft()-&gt;myPlugin_myElement-&gt;saveMyElement($element);
  
  }
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-13T08:48:01.743Z
Comments.Text: Hey @ChadW I've just put some code up there - again thanks for any help you can offer.
Comments.UserDisplayName: Josh Angell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-13T13:21:48.123Z
Comments.Text: I'll dig into this more later, but one thing that stands out is unset() doesn't necessarily free up memory, it just frees up the variable, as I understand it anyway. Thanks for posting your code.
Comments.UserDisplayName: Chad W
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-13T13:22:53.920Z
Comments.Text: Yeah thats what I thought - it did make a small difference in my testing, but then so did just setting it to `null`.
Comments.UserDisplayName: Josh Angell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-12T16:04:53.447Z
Comments.Text: If you wouldn't mind posting some code, it might help us see if there is something that is amiss.
Comments.UserDisplayName: Chad W
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-12T16:06:39.430Z
Comments.Text: Also, I don't know if you've tried working with the `phpMaxMemoryLimit` config setting, although that might just be a bandaid in this case... http://buildwithcraft.com/docs/config-settings#phpMaxMemoryLimit
Comments.UserDisplayName: Chad W
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-12T16:21:00.810Z
Comments.Text: Thanks @ChadW - yes I've set that config setting to the same as my php.ini one and I will try and get some pared down code samples up tonight / tomorrow. Thanks for taking a look!
Comments.UserDisplayName: Josh Angell
Tags: <plugin-development><memory>
Title: Running out of memory in my Task-based plugin
ThreadText: # Running out of memory in my Task-based plugin
  
  (Asked by: username_0 on 2015-02-12)
  
  I am writing a plugin that imports a large amount of data form an API into some custom element types. I have it all set up and working nicely with Tasks however I am running into a fairly big issue with memory.
  
  The plugin runs in the following way:
  
  1. Controlling Task gets set off.
  2. It sets off a SubTask twice that gets two different sets of data from the API and saves it to disk.
  3. It then sets off another bunch of SubTasks that import that data into my custom element types one element at a time - the number of Tasks at this point depends on how large the saved API data is as I split it into chunks of 100.
  
  In my testing I am skipping point 2 from above, so it is only reading stuff from disk, importing it and repeating. I am finding that no matter what I do I can't stop the memory usage building up. I have used `unset()` in various places which has helped a little bit and have turned off the `cacheElementQueries` config parameter using `craft()->config->set('cacheElementQueries', false);` which has also helped but I'm still running into problems.
  
  **Some numbers**
  
  * Resulting files from the API are 2.5MB and 5.3MB of json.
  * Total elements to import from each are 1200 and 2000 respectively.
  * As mentioned above I'm splitting the sets of data in to chunks of 100 to try and conserve memory.
  * php.ini `memory_limit` setting is 512M
  
  **Conclusion**
  
  I feel like I'm missing something obvious as php doesn't dump the memory until all the Tasks have run, or at least that is what it seems like. I kind of thought that after each Task or SubTask had completed the memory for that would be dumped.
  
  I have to confess I'm still a bit new to serious programming and am only using `htop` to monitor the process and can see it going up continually until it reaches my php.ini `memory_limit` setting. I know I should probably use xdebug or something but that kind of scares me...
  
  I've also read this: [Memory issue when updating all entries from a channel (allowed memory size of X bytes exhausted (tried to allocate Y bytes))](https://craftcms.stackexchange.com/questions/740/memory-issue-when-updating-all-entries-from-a-channel-allowed-memory-size-of-x)
  
  If anyone has any ideas however wacky to help with plugging my leaky ship I'd be most grateful! I can post up code samples if it will help but I didn't want to make this post too lengthy.
  
  ---
  
  ## Update
  
  Here is a cut-down set of functions that are used in the process, I've not included lots of little bits that are either just doing processing on non-large variables or other mundane tasks.
  
  **Main Task**
  
  ```php
  public function runStep($step)
  {
  
      // Sync the stored API data to the Craft elements
      return $this->runSubTask('MyPlugin_Sync', null, array(
          'apiElement'         => $this->_apiElements[$step]['apiElement'],
          'apiElementSettings' => $this->_apiElements[$step]['apiElementSettings'],
          'apiService'         => $service,
          'offset'             => $this->_apiElements[$step]['offset']
      ));
  
  }
  
  ```
  
  **Sync Task**
  
  ```php
  public function getTotalSteps()
  {
  
      $this->_apiData = craft()->myPlugin_ApiService->getElementsFromFile($this->_apiElement, $this->_offset, 100);
  
      // Check we got something
      if ( is_array($this->_apiData) )
      {
          return count($this->_apiData);
      }
      else
      {
          return 0;
      }
  
  }
  
  public function runStep($step)
  {
  
      return craft()->myPlugin_ImportService->importElement($this->_apiService, $this->_apiElementSettings, array_shift($this->_apiData));
  
  }
  
  ```
  
  **API Service**
  
  ```php
  public function getElementsFromFile($apiElement, $offset = 0, $limit = null)
  {
  
      // Get the file
      $filename = $this->_storage . $apiElement . 's.json';
  
      $file = IOHelper::getFile($filename);
  
      $data = json_decode($file->getContents(), true);
  
      // Try and help the memory
      unset($file);
  
      // Now select the right portion from it using our offset amount
      return array_slice($data, $offset, $limit);
  
  }
  
  public function mapData($data, $settings)
  {
  
      $return = array();
  
      // Boring mapping code using $data
  
      unset($data);
  
      return $return;
  
  }
  
  ```
  
  **Import Service**
  
  ```php
  public function importElement($apiService, $apiElementSettings, $apiData)
  {
  
      // using the API service, map the data to our fields/attributes held in the plugin settings
      $mappedData = craft()->myPlugin_ApiService->mapData($apiData, $apiElementSettings);
  
      // We’re done with this now, save our memory
      unset($apiData);
  
      // Standard code for getting an existing element and updating it
      // or creating a new one if none exist
  
      // Save the thing
      return craft()->myPlugin_myElement->saveMyElement($element);
  
  }
  
  ```
  
  **Tags:** plugin-development, memory
  
  > **Comments:**
  > **username_1**: If you wouldn't mind posting some code, it might help us see if there is something that is amiss.  
  > **username_1**: Also, I don't know if you've tried working with the `phpMaxMemoryLimit` config setting, although that might just be a bandaid in this case... http://buildwithcraft.com/docs/config-settings#phpMaxMemoryLimit  
  > **username_0**: Thanks @username_1 - yes I've set that config setting to the same as my php.ini one and I will try and get some pared down code samples up tonight / tomorrow. Thanks for taking a look!  
  > **username_0**: Hey @username_1 I've just put some code up there - again thanks for any help you can offer.  
  > **username_1**: I'll dig into this more later, but one thing that stands out is unset() doesn't necessarily free up memory, it just frees up the variable, as I understand it anyway. Thanks for posting your code.  
  > **username_0**: Yeah thats what I thought - it did make a small difference in my testing, but then so did just setting it to `null`.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2015-12-15. Score: 6 | Accepted answer
  
  OK, finally answering my own question here as it seems to not be an issue with the latest Craft 2.5 builds when `devMode` is turned off, but is still an issue when `devMode` is on.
  
  As far as I can tell its something to do with the Craft logging tool spiralling out of control, but thats as far as I got.
  
  > **Comments:**
  > **username_2**: Just to say was having similar issues until disabling dev mode  
  > **username_0**: Yeah took me ages to work out!  
  > **username_4**: Also want to echo these guys, disabling dev mode fixed my memory issue with my tasks plugin. @username_0 have you contacted P&T about this?  
  > **username_0**: @username_4 yeah I have an open ticket with Brad, I think as its not likely to creep into production (or shouldn't rather) and its a nightmare to test against it's not going to be sorted any time soon. I had to profile it to find it.  
  > **username_3**: https://craftcms.com/support/dev-mode This should cast some light on dev mode. Explains what is being logged and more  
  
  ---
  
  