title: Memory for python.exe on Windows 7 python 32 - Numpy uses half of the available memory only?
body: <p>I'm loading large h5 files into memory using <code>numpy</code> <code>ndarray</code>'s. I read that my system (Win 7 prof., 6 GB RAM) is supposed to allow python.exe to use about 2 GB of physical memory.</p>
  
  <p>However I'm getting a <code>MemoryError</code> already just shy of 1 GB. Even stranger this lower limit seems to only apply for <code>numpy array</code>'s but not for a <code>list</code>.</p>
  
  <p>I've tested my memory consumption using the following function found <a href="http://fa.bianp.net/blog/2013/different-ways-to-get-memory-consumption-or-lessons-learned-from-memory_profiler/" rel="nofollow">here</a>:</p>
  
  <pre><code>import psutil
  import gc
  import os
  import numpy as np
  from matplotlib.pyplot import pause
  
  def memory_usage_psutil():
      # return the memory usage in MB
      process = psutil.Process(os.getpid())
      mem = process.get_memory_info()[0]/float(2**20)
      return mem
  </code></pre>
  
  <p><strong>Test 1:</strong> Testing memory limits for an ordinary list</p>
  
  <pre><code>print 'Memory - %d MB' %memory_usage_psutil() # prints memory usage after imports
  a = []
  while 1:
      try:
          a.append([x*2000 for x in xrange(10000)])
      except MemoryError:
          print 'Memory - %d MB' %memory_usage_psutil()
          a = []
          print 'Memory - %d MB' %memory_usage_psutil()
          print 'run garbage collector: collected %d objects.' %gc.collect()
          print 'Memory - %d MB\n\n' %memory_usage_psutil()
          break
  </code></pre>
  
  <p>Test 1 prints:</p>
  
  <pre><code>Memory - 39 MB
  Memory - 1947 MB
  Memory - 1516 MB
  run garbage collector: collected 0 objects.
  Memory - 49 MB
  </code></pre>
  
  <p><strong>Test 2:</strong> Creating a number of large <code>np.array</code>'s</p>
  
  <pre><code>shape = (5500,5500)
  names = ['b', 'c', 'd', 'g', 'h']
  
  try:
      for n in names:
          globals()[n] = np.ones(shape, dtype='float64')
          print 'created variable %s with %0.2f MB'\
          %(n,(globals()[n].nbytes/2.**20))
  except MemoryError:
      print 'MemoryError, Memory - %d MB. Deleting files..'\
      %memory_usage_psutil()
      pause(2)
      # Just added the pause here to be able to observe
      # the spike of memory in the Windows task manager.
      for n in names:
          globals()[n] = []
      print 'Memory - %d MB' %memory_usage_psutil()
      print 'run garbage collector: collected %d objects.' %gc.collect()
      print 'Memory - %d MB' %memory_usage_psutil()
  </code></pre>
  
  <p>Test 2 prints:</p>
  
  <pre><code>Memory - 39 MB
  created variable b with 230.79 MB
  created variable c with 230.79 MB
  created variable d with 230.79 MB
  created variable g with 230.79 MB
  MemoryError, Memory - 964 MB. Deleting files..
  Memory - 39 MB
  run garbage collector: collected 0 objects.
  Memory - 39 MB
  </code></pre>
  
  <p>My question: Why do I get a <code>MemoryError</code> before I'm even close to the 2GB limit and why is there a difference in memory limits for a <code>list</code> and <code>np.array</code> respectively or what am I missing?
  I'm using python 2.7 and numpy 1.7.1</p>