title: Resource use in file read/write with python, options and considerations
body: <p>I'm developing in python, still new to the game, and I want to make sure I tackle this problem correctly.  I'll gladly accept all advice.</p>
  
  <p>Imagine trying to use data stored across multiple flat files, in many cases with a combined size exceeding 20-35 gb.  In the most general case, these files may be delimited (csv, tab) or may simply be fixed width. The goal is to take these files, or some subset of each file, parse the input (each column represents a variable in data), and send it to some destination (which could be local or remote sql, some other local file in a variety of output formats including text or some proprietary data format such as stata's .dta) </p>
  
  <p>The goal is to use available system resources to conduct this operation in the quickest way possible ( i guess in terms of kb per second? )</p>
  
  <p>Questions:</p>
  
  <ol>
  <li><p>Is there an efficiency gain from using compiled C to do the read operations? If so, which library should I learn how to use? And, should c also do the parsing and output?</p></li>
  <li><p>If the file comes in a .zip or .gz, should the entire file be decompressed prior to reading and writing, or do you leave it compressed and use a library capable of reading from the compressed file? </p></li>
  <li><p>Should the program use multi-threading?  I imagine reading some subset of a file (say, n lines at a time), parsing and outputting in, say, j threads.  It's not obvious that reading the file one line at a time is optimal... and it seems like the optimal number of threads and processes will depend upon the resources available and the size of the job.<br>
  So, maybe the code should be 'smart' enough to optimally determine how many threads to use and how much work each thread should be expected to do. </p></li>
  <li><p>How does one measure and compare efficiency between different methods?  </p></li>
  <li><p>Should the program be able to do this dynamically and choose input output methods based on their performance? (Will method A always strictly dominate method B, or do idiosyncratic changes in the deployment environment matter) </p></li>
  </ol>
  
  <p>Just to be clear, I'm willing to accept almost any level of code inefficiency in exchange for non-negligible improvements in resource efficiency </p>
  
  <p>I hope these questions provide a clear idea of what I'm trying to figure out.  My programming experience is mostly confined to scientific/statistical packages, so if any of my questions boil down to 'rtm', please be gentle and suggest the appropriate manual. </p>