title: Python Multiprocessing concurrency using Manager, Pool and a shared list not working
body: <p>I am learning python multiprocessing, and I am trying to use this feature to populate a list with all the files present in an os. However, the code that I wrote is executing sequentially only.</p>
  
  <pre><code>#!/usr/bin/python
  import os
  import multiprocessing
  tld = [os.path.join("/", f) for f in os.walk("/").next()[1]] #Gets a top level directory names inside "/"
  manager = multiprocessing.Manager()
  files = manager.list()
  
  
  def get_files(x):
      for root, dir, file in os.walk(x):
          for name in file:
              files.append(os.path.join(root, name))
  
  mp = [multiprocessing.Process(target=get_files, args=(tld[x],))
        for x in range(len(tld))]
  
  for i in mp:
      i.start()
      i.join()
  print len(files)
  </code></pre>
  
  <p>When I checked the process tree, I can see only a single chile processes spawned. (man pstree says {} denotes the child process spawned by the parent.)</p>
  
  <pre><code>---bash(10949)---python(12729)-+-python(12730)---{python}(12752)
                                 `-python(12750)`
  </code></pre>
  
  <p>What I was looking for was, to spawn a process for each tld directory, populate the shared list <code>files</code>, and that would be around 10-15 processes depending on the number of directories. What am I doing wrong?</p>
  
  <p>EDIT:: </p>
  
  <p>I used <code>multiprocessing.Pool</code> to create worker threads, and this time the 
  processes are spawned, but is giving errors when I try to use<code>multiprocessing.Pool.map()</code>. I was referring to the following code in python docs that shows</p>
  
  <pre><code>from multiprocessing import Pool
  def f(x):
  return x*x
  
  if __name__ == '__main__':
      p = Pool(5)
      print(p.map(f, [1, 2, 3])) 
  </code></pre>
  
  <p>Following that example, I rewrote the code as</p>
  
  <pre><code>import os
  import multiprocessing
  tld = [os.path.join("/", f) for f in os.walk("/").next()[1]]
  manager = multiprocessing.Manager()
  pool = multiprocessing.Pool(processes=len(tld))
  print pool
  files = manager.list()
  def get_files(x):
      for root, dir, file in os.walk(x):
          for name in file:
              files.append(os.path.join(root, name))
  pool.map(get_files, [x for x in tld])
  pool.close()
  pool.join()
  print len(files)
  </code></pre>
  
  <p>and it is forking multiple processes.</p>
  
  <pre><code>---bash(10949)---python(12890)-+-python(12967)
                                 |-python(12968)
                                 |-python(12970)
                                 |-python(12971)
                                 |-python(12972)
                                 ---snip---
  </code></pre>
  
  <p>But the code is erroring saying</p>
  
  <p><code>Process PoolWorker-2:
  Traceback (most recent call last):
    File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  Traceback (most recent call last):
    File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      self.run()
    File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      task = get()
    File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
      return recv()
  AttributeError: 'module' object has no attribute 'get_files'
      self._target(*self._args, **self._kwargs)
      self.run()
      task = get()
    File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      self.run()
    File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      task = get()
    File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
  AttributeError: 'module' object has no attribute 'get_files'
      self.run()
  </code></p>
  
  <p>What am I doing wrong here, and why do the get_files() function errors out?</p>