title: Multiprocessing writing to pandas dataframe
body: <p>So what I am trying to do with the following code is to read a list of lists and put them through function called <code>checker</code> and then have <code>log_result</code> deal with the result of the function <code>checker</code>. I am trying to do this using multithreading because the variable name <code>rows_to_parse</code> in reality has millions of rows, so using multiple cores should speed up this process by a considerable amount. </p>
  
  <p>The code at present moment doesn't work and crashes Python.</p>
  
  <p>Concerns and Issues I have:</p>
  
  <ol>
  <li>Want the existing df which held in the variable <code>df</code> to maintain the
  index throughout process because otherwise <code>log_result</code> will get
  confused as to which row needs updating.</li>
  <li>I am quite certain that <code>apply_async</code> is not the appropriate
  multiprocessing function to perform this duty because I believe the
  order at which the computer reads and writes the df can possibly corrupt it???  </li>
  <li>I think that a queue may need to be set up to write and read <code>df</code>
  but I am unsure as to how I would go about doing that.</li>
  </ol>
  
  <p>Thank you for any assistance.</p>
  
  <pre><code>import pandas as pd
  import multiprocessing
  from functools import partial
  
  def checker(a,b,c,d,e):
      match = df[(df['a'] == a) &amp; (df['b'] == b) &amp; (df['c'] == c) &amp; (df['d'] == d) &amp; (df['e'] == e)]
      index_of_match = match.index.tolist()
      if len(index_of_match) == 1: #one match in df
          return index_of_match
      elif len(index_of_match) &gt; 1: #not likely because duplicates will be removed prior to: if "__name__" == __main__:
          return [index_of_match[0]]
      else: #no match, returns a result which then gets processed by the else statement in log_result. this means that [a,b,c,d,e] get written to the df
          return [a,b,c,d,e]
  
  
  
  def log_result(result, dataf):
      if len(result) == 1: #
          dataf.loc[result[0]]['e'] += 1 
      else: #append new row to exisiting df
          new_row = pd.DataFrame([result],columns=cols)
          dataf = dataf.append(new_row,ignore_index=True)
  
  
  def apply_async_with_callback(parsing_material, dfr):
      pool = multiprocessing.Pool()
      for var_a, var_b, var_c, var_d, var_e in parsing_material:
          pool.apply_async(checker, args = (var_a, var_b, var_c, var_d, var_e), callback = partial(log_result,dataf=dfr))
      pool.close()
      pool.join()
  
  
  
  if __name__ == '__main__':
      #setting up main dataframe
      cols = ['a','b','c','d','e']
      existing_data = [["YES","A","16052011","13031999",3],
                      ["NO","Q","11022003","15081999",3],
                      ["YES","A","22082010","03012001",9]]
  
      #main dataframe
      df = pd.DataFrame(existing_data,columns=cols)
  
      #new data
      rows_to_parse = [['NO', 'A', '09061997', '06122003', 5],
                      ['YES', 'W', '17061992', '26032012', 6],
                      ['YES', 'G', '01122006', '07082014', 2],
                      ['YES', 'N', '06081992', '21052008', 9],
                      ['YES', 'Y', '18051995', '24011996', 6],
                      ['NO', 'Q', '11022003', '15081999', 3],
                      ['NO', 'O', '20112004', '28062008', 0],
                      ['YES', 'R', '10071994', '03091996', 8],
                      ['NO', 'C', '09091998', '22051992', 1],
                      ['YES', 'Q', '01051995', '02012000', 3],
                      ['YES', 'Q', '26022015', '26092007', 5],
                      ['NO', 'F', '15072002', '17062001', 8],
                      ['YES', 'I', '24092006', '03112003', 2],
                      ['YES', 'A', '22082010', '03012001', 9],
                      ['YES', 'I', '15072016', '30092005', 7],
                      ['YES', 'Y', '08111999', '02022006', 3],
                      ['NO', 'V', '04012016', '10061996', 1],
                      ['NO', 'I', '21012003', '11022001', 6],
                      ['NO', 'P', '06041992', '30111993', 6],
                      ['NO', 'W', '30081992', '02012016', 6]]
  
  
      apply_async_with_callback(rows_to_parse, df)
  </code></pre>