title: Processing a byte string in chunks
body: <p>I have a really long byte string such as this (the actual value may be random):</p>
  <pre class="lang-py prettyprint-override"><code>in_var = b'\x01\x02\x03\x04\x05\x06...\xff'
  </code></pre>
  <p>I also have a function that performs an operation on a chunk of bytes and returns the same number of bytes (let's say 10 bytes for this example):</p>
  <pre class="lang-py prettyprint-override"><code>def foo(chunk):
    # do smth with chunk
    # ......
    return chunk
  </code></pre>
  <p>I want to process <code>in_var</code> with <code>foo()</code> for all chunks of 10 bytes (sending the last chunk as is if less than 10 bytes remain at the end) and create a new variable <code>out_var</code> with the outputs.</p>
  <p>The way I'm currently doing it is taking way too long:</p>
  <pre class="lang-py prettyprint-override"><code>out_var = b''
  for chunk in range(0, len(in_var), 10):
      out_var += foo(in_var[chunk: chunk + 10])
  </code></pre>
  <p>The function <code>foo()</code> only takes a fraction of a second per run, so the total should be very fast (total of all chunks of 10). However, I'm getting an order of magnitude longer.</p>
  <p>I also tried this with similar results:</p>
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  import math
  in_var= np.array_split(np.frombuffer(in_var, dtype=np.uint8), math.ceil(len(in_var)/10))
  out_var= b&quot;&quot;.join(map(lambda x: foo(x), in_var))
  </code></pre>
  <p><code>foo()</code> can only process 10 bytes for this example (ex: it's an encryption function with a fixed block size) and if a smaller chunk is given to it, it just pads to make the chunk 10 bytes. Let's say I have no control over it, and <code>foo()</code> can only process in chunks of 10 bytes.</p>
  <p>Is there a much faster way to do this? As a last resort, I may have to parallelize my code so all chunks get processed in parallel...</p>
  <p>Thank you!</p>
  <hr />
  <p>UPDATE:</p>
  <p>Apparently I had not correctly measured the time <code>foo()</code> takes. It turns out, <code>foo()</code> is taking the majority of the time, hence the order of magnitude comment above. Thank you again for your comments and suggestions, I did make some improvements nevertheless. Parallelizing the code seems to be the correct path forward.</p>