title: Efficient file reading in python with need to split on '\n'
body: <p>I've traditionally been reading in files with:</p>
  
  <pre><code>file = open(fullpath, "r")
  allrecords = file.read()
  delimited = allrecords.split('\n')
  for record in delimited[1:]:
      record_split = record.split(',')
  </code></pre>
  
  <p>and</p>
  
  <pre><code>with open(os.path.join(txtdatapath,pathfilename), "r") as data:
    datalines = (line.rstrip('\r\n') for line in data)
    for record in datalines:
      split_line = record.split(',')
      if len(split_line) &gt; 1:
  </code></pre>
  
  <p>But it seems when I process these files in a multiprocessing thread I get MemoryError. How can I best readin files line by line, when the text file I'm reading needs to be split on <code>'\n'</code>.</p>
  
  <p>Here is the multiprocessing code:</p>
  
  <pre><code>pool = Pool()
  fixed_args = (targetdirectorytxt, value_dict)
  varg = ((filename,) + fixed_args for filename in readinfiles)
  op_list = pool.map_async(PPD_star, list(varg), chunksize=1)     
  while not op_list.ready():
    print("Number of files left to process: {}".format(op_list._number_left))
    time.sleep(60)
  op_list = op_list.get()
  pool.close()
  pool.join()
  </code></pre>
  
  <p>Here is the error log</p>
  
  <pre><code>Exception in thread Thread-3:
  Traceback (most recent call last):
    File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
      self.run()
    File "C:\Python27\lib\threading.py", line 763, in run
      self.__target(*self.__args, **self.__kwargs)
    File "C:\Python27\lib\multiprocessing\pool.py", line 380, in _handle_results
      task = get()
  MemoryError
  </code></pre>
  
  <p>I'm trying to install pathos as Mike has kindly suggested but I'm running into issues. Here is my install command:</p>
  
  <pre><code>pip install https://github.com/uqfoundation/pathos/zipball/master --allow-external pathos --pre
  </code></pre>
  
  <p>But here are the error messages that I get:</p>
  
  <pre><code>Downloading/unpacking https://github.com/uqfoundation/pathos/zipball/master
    Running setup.py (path:c:\users\xxx\appdata\local\temp\2\pip-1e4saj-b
  uild\setup.py) egg_info for package from https://github.com/uqfoundation/pathos/
  zipball/master
  
  Downloading/unpacking ppft&gt;=1.6.4.5 (from pathos==0.2a1.dev0)
    Running setup.py (path:c:\users\xxx\appdata\local\temp\2\pip_build_jp
  tyuser\ppft\setup.py) egg_info for package ppft
  
      warning: no files found matching 'python-restlib.spec'
  Requirement already satisfied (use --upgrade to upgrade): dill&gt;=0.2.2 in c:\pyth
  on27\lib\site-packages\dill-0.2.2-py2.7.egg (from pathos==0.2a1.dev0)
  Requirement already satisfied (use --upgrade to upgrade): pox&gt;=0.2.1 in c:\pytho
  n27\lib\site-packages\pox-0.2.1-py2.7.egg (from pathos==0.2a1.dev0)
  Downloading/unpacking pyre==0.8.2.0-pathos (from pathos==0.2a1.dev0)
    Could not find any downloads that satisfy the requirement pyre==0.8.2.0-pathos
   (from pathos==0.2a1.dev0)
    Some externally hosted files were ignored (use --allow-external pyre to allow)
  .
  Cleaning up...
  No distributions at all found for pyre==0.8.2.0-pathos (from pathos==0.2a1.dev0)
  
  Storing debug log for failure in C:\Users\xxx\pip\pip.log
  </code></pre>
  
  <p>I'm installing on Windows 7 64 bit. In the end I managed to install with easy_install.</p>
  
  <p>But Now I have a failure as I cannot open that many files:</p>
  
  <pre><code>Finished reading in Exposures...
  Reading Samples from:  C:\XXX\XXX\XXX\
  Traceback (most recent call last):
    File "events.py", line 568, in &lt;module&gt;
      mdrcv_dict = ReadDamages(damage_dir, value_dict)
    File "events.py", line 185, in ReadDamages
      res = thpool.amap(mppool.map, [rstrip]*len(readinfiles), files)
    File "C:\Python27\lib\site-packages\pathos-0.2a1.dev0-py2.7.egg\pathos\multipr
  ocessing.py", line 230, in amap
      return _pool.map_async(star(f), zip(*args)) # chunksize
    File "events.py", line 184, in &lt;genexpr&gt;
      files = (open(name, 'r') for name in readinfiles[0:])
  IOError: [Errno 24] Too many open files: 'C:\\xx.csv'
  </code></pre>
  
  <p>Currently using the multiprocessing library, I am passing in parameters and dictionaries into my function and opening a mapped file and then outputting a dictionary. Here is an example of how I currently do it, how would the smart way to do this with pathos?</p>
  
  <pre><code>def PP_star(args_flat):
      return PP(*args_flat)
  
  def PP(pathfilename, txtdatapath, my_dict):
      return com_dict
  
  fixed_args = (targetdirectorytxt, my_dict)
  varg = ((filename,) + fixed_args for filename in readinfiles)
  op_list = pool.map_async(PP_star, list(varg), chunksize=1)
  </code></pre>
  
  <p>How can I perform the same function with <code>pathos.multiprocessing</code></p>