title: data matching multiple columns with fuzzy matching criteria
body: <p>I'm doing a similar project to this: <a href="https://stackoverflow.com/questions/2435256/data-matching-algorithm">Data matching algorithm</a></p>
  
  <p>Where, I have one data frame (dataset1) with customer details, without a special unique ID, which is then matched against data frame 2 (dataset2) which has the same fields along with a special unique ID.</p>
  
  <p>Example:
  Dataset1 columns include:</p>
  
  <p>Title, Last name,  First name, Middle/other name, address,  DOB, sex, health care number</p>
  
  <p>dataset2 columns include:</p>
  
  <p>Unique ID, Title, Last name,  First name, Middle/other name, address,  DOB, sex, health care number</p>
  
  <p><strong>My plan:</strong></p>
  
  <p>My data set 1 is ~500,000 rows w/ 11 or so columns - the row count can be split if required. Data set 2 is ~2,000,000</p>
  
  <p>Based on matching criteria from the business eg. >5 columns must match, and potentially a fuzzy component eg names that are hyphenated (inc. other special characters) in one data set but not in the other.</p>
  
  <p>Based on the approximate match, this ID, and the fields from data set 2 is copied into data set 1 in adjacent columns, along with an overall match %.</p>
  
  <p>Then overall match % ranges can be extracted and inspected by the business.</p>
  
  <p>I plan on using Panda with Fuzzywuzzy.
  I'm having trouble planning my technical approach on the logic of my code.</p>
  
  <p>SHould I look through the first field for matches, then filter down on the next column, and so on? I think so, but how does the loop operate moving across the columns in the two comparative data sets?
  do I need to write each operation of comparison back into the data set for storage or can it be kept else where?</p>
  
  <p>the alternative that might work, is that I concatenate all the columns into a single column, and those are matched, and returned with the matched ID, other fields and the matching %.</p>
  
  <p>I'm looking for the best direction to head into, along with general logic on the best way to do it. </p>