Answers.Body: <p>Ok, update..</p>
  
  <p>The problem of my high memory usage was caused by (I think the logging of) devmode.. When I set devmode to false the moment that hit 10+gig memory before now has 100mb of memory usage... </p>
  
  <p>Resetting memory on the different tasks that get initiated in the same function or as subtasks is not possible I guess?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-29T13:09:09.740Z
Answers.LastActivityDate: 2019-01-29T13:09:09.740Z
Answers.UserDisplayName: Pieter-Jan De Bruyne
Body: <p>I build a plugin for Craft 2 which imports data from an excel file into craft commerce products.</p>
  
  <p>The problem is that it needs to import excels with like 10.0000 rows and 30 columns.</p>
  
  <p>I made a simple page in the CP where they can upload the file and then view the progress inside of the tasks.</p>
  
  <p>Running just 1 task was not good because if it crashes on the first row it just stops. I wanted to make some kind of batching with tasks so this s what I came up with:</p>
  
  <pre><code>public function startExcelImportTasks($fileName){
      $return = false;
      $dataObject = craft()-&gt;myplugin_productsInfo-&gt;getDataByExcelName($fileName);
      $grandTotal = count((array)$dataObject);
      $batchSize = 500;
      if ($grandTotal &gt; 0) {
          for ($i=0; $i &lt; $grandTotal; $i+=$batchSize) { 
              $numberOfTasks = $batchSize;
              //if last loop set numberoftasks to total left
              if (($i + $batchSize) &gt; $grandTotal){
                  $numberOfTasks = $grandTotal - $i;
              }
              unset($dataObjectSlice, $dataObjectSliceString, $task);
              $dataObjectSlice = array_slice( $dataObject , $i , $numberOfTasks );
              $dataObjectSliceString = json_encode($dataObjectSlice);
              //create a task per batch
              $task = craft()-&gt;tasks-&gt;createTask('Myplugin_ImportExcel', 'Import excel - ('. $i . '-' . ($numberOfTasks + $i) .')', ['totalSteps' =&gt; $numberOfTasks, 'offset' =&gt; $i, 'grandTotal' =&gt; $grandTotal, 'dataObject' =&gt; $dataObjectSliceString]);
          }
          $return = true;
      }
      unset($dataObject,$grandTotal,$batchSize,$numberOfTasks,$dataObjectSlice,$dataObjectSliceString, $task);
      return $return;
  }
  </code></pre>
  
  <p>So this is all running very well, I get tasks split up by 500 rows. My problem is memory usage, I thought when a new task would start it would free up the memory of the previous task but when I log my memory usage with memory_get_usage(true) it does not reset after the first loop and just continues to build up loop after loop until it finally crashes. It then goes to the next step starting with all free memory.</p>
  
  <p>Is there any way to reset the memory for each loop? Loop 2 does not need to know ANYTHING that loop 1 needs since I slice the data and pass it to the right task.</p>
  
  <p>I tried using some unsets in the code but that doesn't seem to do very much.</p>
  
  <p>I also came across some subtask posts but I do not think that will change anything? I also like that the batching is shown in different tasks in the CP.</p>
  
  <p>Any help is welcome,</p>
  
  <p>Thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-29T10:27:55.350Z
Comments.Text: I since tried using a main task with batch sub tasks to run my import but the memory keeps adding up the more subtasks that have already ran (same result as above).
Comments.UserDisplayName: Pieter-Jan De Bruyne
Tags: <plugin-development><import><tasks><memory><batch>
Title: Lowering memory usage on task batches
ThreadText: # Lowering memory usage on task batches
  
  (Asked by: username_0 on 2019-01-24)
  
  I build a plugin for Craft 2 which imports data from an excel file into craft commerce products.
  
  The problem is that it needs to import excels with like 10.0000 rows and 30 columns.
  
  I made a simple page in the CP where they can upload the file and then view the progress inside of the tasks.
  
  Running just 1 task was not good because if it crashes on the first row it just stops. I wanted to make some kind of batching with tasks so this s what I came up with:
  
  ```
  public function startExcelImportTasks($fileName){
      $return = false;
      $dataObject = craft()->myplugin_productsInfo->getDataByExcelName($fileName);
      $grandTotal = count((array)$dataObject);
      $batchSize = 500;
      if ($grandTotal > 0) {
          for ($i=0; $i < $grandTotal; $i+=$batchSize) { 
              $numberOfTasks = $batchSize;
              //if last loop set numberoftasks to total left
              if (($i + $batchSize) > $grandTotal){
                  $numberOfTasks = $grandTotal - $i;
              }
              unset($dataObjectSlice, $dataObjectSliceString, $task);
              $dataObjectSlice = array_slice( $dataObject , $i , $numberOfTasks );
              $dataObjectSliceString = json_encode($dataObjectSlice);
              //create a task per batch
              $task = craft()->tasks->createTask('Myplugin_ImportExcel', 'Import excel - ('. $i . '-' . ($numberOfTasks + $i) .')', ['totalSteps' => $numberOfTasks, 'offset' => $i, 'grandTotal' => $grandTotal, 'dataObject' => $dataObjectSliceString]);
          }
          $return = true;
      }
      unset($dataObject,$grandTotal,$batchSize,$numberOfTasks,$dataObjectSlice,$dataObjectSliceString, $task);
      return $return;
  }
  
  ```
  
  So this is all running very well, I get tasks split up by 500 rows. My problem is memory usage, I thought when a new task would start it would free up the memory of the previous task but when I log my memory usage with memory\_get\_usage(true) it does not reset after the first loop and just continues to build up loop after loop until it finally crashes. It then goes to the next step starting with all free memory.
  
  Is there any way to reset the memory for each loop? Loop 2 does not need to know ANYTHING that loop 1 needs since I slice the data and pass it to the right task.
  
  I tried using some unsets in the code but that doesn't seem to do very much.
  
  I also came across some subtask posts but I do not think that will change anything? I also like that the batching is shown in different tasks in the CP.
  
  Any help is welcome,
  
  Thanks
  
  **Tags:** plugin-development, import, tasks, memory, batch
  
  > **Comments:**
  > **username_0**: I since tried using a main task with batch sub tasks to run my import but the memory keeps adding up the more subtasks that have already ran (same result as above).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-01-29. Score: 2 | Accepted answer
  
  Ok, update..
  
  The problem of my high memory usage was caused by (I think the logging of) devmode.. When I set devmode to false the moment that hit 10+gig memory before now has 100mb of memory usage...
  
  Resetting memory on the different tasks that get initiated in the same function or as subtasks is not possible I guess?
  
  
  ---
  
  