title: Updating database with callback in Parallel Python
body: <p>I'm trying to do some text processing on around 200,000 entries in a SQlite database which I'm accessing using SQLAlchemy.  I'd like to parallelize it (I'm looking at Parallel Python), but I'm not sure how exactly to do it.</p>
  
  <p>I want to commit the session each time an entry is processed, so that if I need to stop the script I won't lose the work it's already done.  However, when I try to pass the session.commit() command to the callback function, it does not seem to work.</p>
  
  <pre><code>from assignDB import *
  from sqlalchemy.orm import sessionmaker
  import pp, sys, fuzzy_substring
  
  def matchIng(rawIng, ingreds):
  maxScore = 0
  choice = ""
  for (ingred, parentIng) in ingreds.iteritems():
      score = len(ingred)/(fuzzy_substring(ingred,rawIng)+1)
      if score &gt; maxScore:
          maxScore = score
          choice = ingred
          refIng = parentIng  
  return (refIng, choice, maxScore)
  
  def callbackFunc(match, session, inputTuple):
      print inputTuple
      match.refIng_id = inputTuple[0]
      match.refIng_name = inputTuple[1]
      match.matchScore = inputTuple[2]
      session.commit()
  
  # tuple of all parallel python servers to connect with
  ppservers = ()
  #ppservers = ("10.0.0.1",)
  
  if len(sys.argv) &gt; 1:
      ncpus = int(sys.argv[1])
      # Creates jobserver with ncpus workers
      job_server = pp.Server(ncpus, ppservers=ppservers)
  else:
      # Creates jobserver with automatically detected number of workers
      job_server = pp.Server(ppservers=ppservers)
  
  print "Starting pp with", job_server.get_ncpus(), "workers"
  
  ingreds = {}
  for synonym, parentIng in session.query(IngSyn.synonym, IngSyn.parentIng): 
      ingreds[synonym] = parentIng
  
  jobs = []
  for match in session.query(Ingredient).filter(Ingredient.refIng_id == None):
      rawIng = match.ingredient
      jobs.append((match, job_server.submit(matchIng,(rawIng,ingreds),    (fuzzy_substring,),callback=callbackFunc,callbackargs=(match,session))))
  </code></pre>
  
  <p>The session is imported from <code>assignDB</code>.  I'm not getting any error, it's just not updating the database.</p>
  
  <p>Thanks for your help.</p>
  
  <p><strong>UPDATE</strong>
  Here is the code for fuzzy_substring</p>
  
  <pre><code>def fuzzy_substring(needle, haystack):
      """Calculates the fuzzy match of needle in haystack,
      using a modified version of the Levenshtein distance
      algorithm.
      The function is modified from the levenshtein function
      in the bktree module by Adam Hupp"""
      m, n = len(needle), len(haystack)
  
      # base cases
      if m == 1:
          return not needle in haystack
      if not n:
          return m
  
      row1 = [0] * (n+1)
      for i in range(0,m):
          row2 = [i+1]
          for j in range(0,n):
              cost = ( needle[i] != haystack[j] )
  
              row2.append( min(row1[j+1]+1, # deletion
                                 row2[j]+1, #insertion
                                 row1[j]+cost) #substitution
                             )
          row1 = row2
      return min(row1)
  </code></pre>
  
  <p>which I got from here: <a href="http://ginstrom.com/scribbles/2007/12/01/fuzzy-substring-matching-with-levenshtein-distance-in-python/" rel="nofollow">Fuzzy Substring</a>.  In my case, "needle" is one of ~8000 possible choices, while haystack is the raw string I'm trying to match.  I loop over all possible "needles" and choose the one with the best score.  </p>