title: Attempting to optimize the following loop over numpy arrays? Best method ? (numba or dask)
body: <p>I am trying to refactor this code in order to minimize its runtime and memory usage (if possible)</p>
  <pre><code>for i in range(gbl.NumStoreRows):
      cal_effects[i,:,:len(orig_cols)] = cal_effects_vals - **Use ~1gb memory on this line**
      priors[i,:len(orig_cols)] = orig_prior_coeffs
      priors_SE[i,:len(orig_cols)] = orig_prior_SE
  </code></pre>
  <p>It is only the first operation in the loop which is time/memory intensive, I tried splitting the the memory/runtime intensive line from the other two and created two separate loops. - just made it a second slower, and no memory impact.</p>
  <p>I tried to create a jit function for this code block then, but the application stops running later on in the code with error message. - It just stops on one of the LoadFunctions(), so I think jit might be altering the output or my function is incorrectly structured.</p>
  <p><strong>Variations of my jit function</strong>
  <strong>Variation 1</strong>
  @jit
  def populate_cal_effects(cal_effects_vals):
  for i in range(gbl.NumStoreRows):
  cal_effects[i,:,:len(orig_cols)] = cal_effects_vals</p>
  <pre><code>populate_cal_effects(cal_effects_vals)
  
  for i in range(gbl.NumStoreRows):
      priors[i,:len(orig_cols)] = orig_prior_coeffs
      priors_SE[i,:len(orig_cols)] = orig_prior_SE
  </code></pre>
  <p><strong>Variation 2: Adding a return statement to the function</strong></p>
  <pre><code>@jit
  def populate_cal_effects(cal_effects_vals):
      for i in range(gbl.NumStoreRows):
          cal_effects[i,:,:len(orig_cols)] = cal_effects_vals
      return  cal_effects[i,:,:len(orig_cols)]
  </code></pre>
  <p><strong>Variation 3: add the operations from the other for loop to the function</strong></p>
  <ul>
  <li><p>This was the method I expected to be fastest and not affect data output</p>
  <pre><code>  @jit(parallel=True)
    def populate_cal_effects(cal_effects_vals):
    for i in prange(gbl.NumStoreRows):
        cal_effects[i,:,:len(orig_cols)] = cal_effects_vals
        priors[i,:len(orig_cols)] = orig_prior_coeffs
        priors_SE[i,:len(orig_cols)] = orig_prior_SE
  </code></pre>
  </li>
  <li><p>I wanted to utilize parallel mode and use prange for the loop, but I cannot get this to work.</p>
  </li>
  </ul>
  <p><strong>Context/Other:</strong></p>
  <ul>
  <li>I have defined this function inside the main load function. - My next step is too move it out of the Load function and re-run.</li>
  <li>If this method doesn't work I was thinking of trying to process in parallel (multiple cores) - not machines. using Dask.</li>
  </ul>
  <p><em>Any pointers on this would be great, maybe I am wasting my time and this is not optimizable, if so, do let me know</em></p>
  <p><strong>Steps to reproduce</strong></p>
  <ul>
  <li><p>gbl.NumstoreRows = 866 (# of stores)</p>
  </li>
  <li><p>All data types are numpy arrays</p>
  <p><code>cal_effects = np.zeros((gbl.NumStoreRows, n_days, n_cal_effects), dtype=np.float64)</code></p>
  <p><code>priors = np.zeros((gbl.NumStoreRows, n_cal_effects), dtype=np.float64)</code></p>
  <p><code>priors_SE = np.zeros((gbl.NumStoreRows, n_cal_effects), dtype=np.float64)</code></p>
  </li>
  </ul>