title: Why does numpy.astype throw a memory error when the nparray size is below the memory limit?
body: <pre><code>import numpy as np
  N = 12500
  x = np.random.randint(100, size=(N,N)) # int32
  x = x.astype('int16', copy=False)
  </code></pre>
  
  <p>throws</p>
  
  <blockquote>
    <p>numpy.core._exceptions.MemoryError: Unable to allocate array with shape (12500, 12500) and data type int16</p>
  </blockquote>
  
  <p>For some reason <code>np.random.randint(100, size=(N,N),dtype='int16')</code> works. The problem is that I have to load an npz compressed file (which I represent as a random nparray here) and then convert from int32 to int16.</p>
  
  <p>I used memory_profiler to check it out, running <code>python -m memory_profiler ./testProc.py</code></p>
  
  <pre><code>import numpy as np
  @profile
  def test_size(N=12500):
      x = np.random.randint(100, size=(N,N))
      x = x.astype('int16', copy=False)
      del(x)
  
  if __name__ == '__main__':
      test_size()
  </code></pre>
  
  <p>yielding</p>
  
  <pre><code>Line #    Mem usage    Increment   Line Contents
  ================================================
       2   24.648 MiB   24.648 MiB   @profile
       3                             def test_size(N=2500*5):
       4  620.781 MiB  596.133 MiB       x = np.random.randint(100, size=(N,N))
       5  620.789 MiB    0.008 MiB       print(x.shape)
       6  620.789 MiB    0.000 MiB       print(x.dtype)
       7  620.793 MiB    0.004 MiB       x = x.astype('int16', copy=False)
       8                                 del(x)
  </code></pre>
  
  <p>with the same error as before (hence the conversion in line 7 fails).</p>
  
  <p>As far as I can tell, I should not have a memory problem. The array (12500 by 12500) is roughly 600 MB as int32 and 300 MB as int16. So I am not even close to the ~2 GB limit of Python 32 bit on Windows 10. Unless there is some internal conversion to float64/int64 before converting to int16? Or something along that line. I have 32 GB ram installed, so that should not be the culprit.</p>
  
  <p>I am aware that I can slice the data and/or switch to Python 64 bit - but what's up with this memory problem when I am far from the memory limit (at least as far as I can tell)?</p>
  
  <p>Found this: <a href="https://stackoverflow.com/questions/57748040/how-to-fix-numpy-core-exceptions-memoryerror-while-performing-mnist-digit-cla">How to fix &quot;numpy.core._exceptions.MemoryError&quot; while performing MNIST digit classifier?</a> , where it seems like switching to Linux solves the problem (not an option for me). Is it a Windows problem? Something along the lines of contiguous memory allocation?</p>
  
  <p>Also found this <a href="https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type">Unable to allocate array with shape and data type</a>. Could it be some overcommit handling problem in Windows?</p>
  
  <p>Thanks for any help.</p>