Body: <p>I'm new to AI/ML and I want to research and learn about techniques that could help me to solve this complex task. Any hint would be appreciated.</p>
  <p>Let me explain it with an example:</p>
  <p>Let's look at two columns <code>PUR.SUPPLY.MTL_REQ_HDR_ID</code> and <code>MTL.PO_REQUISITION_HEADERS_TAB.ID</code>. It is likely they are related (one is the FK and the other one is PK).</p>
  <p><a href="https://i.sstatic.net/etgPH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/etgPH.png" alt="enter image description here" /></a></p>
  <p>As a human I was able to do it by doing the following:</p>
  <ol>
  <li>I decoded abbreviations,</li>
  <li>I identified context (MTL module),</li>
  <li>I identified the subject (requisition header),</li>
  <li>I identified ID keyword,</li>
  <li>I identified irrelevant information (TAB postfix),</li>
  <li>I matched words that are not in the exact order,</li>
  <li>I estimated which elements/words do not have to match/occur,</li>
  </ol>
  <p>I would like to match millions of columns relatively quickly (seconds). I would like algorithm to learn:</p>
  <ol>
  <li>what words are likely context,</li>
  <li>what are irrelevant,</li>
  <li>what are subjects,</li>
  <li>ideally learn some patterns (prefixes, postfixes, name formats, etc),</li>
  <li>based on user responses - approve/reject match,</li>
  <li>build dictionary of abbreviations,</li>
  <li>estimate probability...</li>
  </ol>
  <p>I know this is a complex task, but maybe you know a technique, tool, library, article, example ... anything that could be helpful? Any help would be appreciated.</p>
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-13T04:06:30.350Z
Comments.Text: In general Bayesian networks/Naive Bayes do well at text processing. As a library sci-kit has an implementation.
Comments.UserDisplayName: A_Arnold
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-13T18:42:35.857Z
Comments.Text: Thanks @A_Arnold, Naive Bayes seems to be a classification function. I don't think my problem is a classificaiton problem.
Comments.UserDisplayName: Bad Pitt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-13T20:23:02.593Z
Comments.Text: If you already have some of these matches you can try and scale it to new data. i.e. your pk & fk cols would be x1, x2, match would be y1=1 no match y1=0. Do this for some data, then let it predict some data, update that data if the classification is wrong etc. ALT: you may want to look into ICA which can predict topics. https://www.cc.gatech.edu/~isbell/papers/isbell-ica-nips-1999.pdf
Comments.UserDisplayName: A_Arnold
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-14T21:34:47.640Z
Comments.Text: thanks @A_Arnold. We will need to test the bayes and research the ICA. That might be a good start.
Comments.UserDisplayName: Bad Pitt
Tags: <machine-learning><natural-language-understanding>
Title: Text matching: fuzzy names matching with learning
UserDisplayName: Bad Pitt
ThreadText: # Text matching: fuzzy names matching with learning
  
  (Asked by: username_0 on 2021-06-12)
  
  I'm new to AI/ML and I want to research and learn about techniques that could help me to solve this complex task. Any hint would be appreciated.
  
  Let me explain it with an example:
  
  Let's look at two columns `PUR.SUPPLY.MTL_REQ_HDR_ID` and `MTL.PO_REQUISITION_HEADERS_TAB.ID`. It is likely they are related (one is the FK and the other one is PK).
  
  [![enter image description here](https://i.sstatic.net/etgPH.png)](https://i.sstatic.net/etgPH.png)
  
  As a human I was able to do it by doing the following:
  
  1. I decoded abbreviations,
  2. I identified context (MTL module),
  3. I identified the subject (requisition header),
  4. I identified ID keyword,
  5. I identified irrelevant information (TAB postfix),
  6. I matched words that are not in the exact order,
  7. I estimated which elements/words do not have to match/occur,
  
  I would like to match millions of columns relatively quickly (seconds). I would like algorithm to learn:
  
  1. what words are likely context,
  2. what are irrelevant,
  3. what are subjects,
  4. ideally learn some patterns (prefixes, postfixes, name formats, etc),
  5. based on user responses - approve/reject match,
  6. build dictionary of abbreviations,
  7. estimate probability...
  
  I know this is a complex task, but maybe you know a technique, tool, library, article, example ... anything that could be helpful? Any help would be appreciated.
  
  Thanks.
  
  **Tags:** machine-learning, natural-language-understanding
  
  > **Comments:**
  > **username_1**: In general Bayesian networks/Naive Bayes do well at text processing. As a library sci-kit has an implementation.  
  > **username_0**: Thanks @username_1, Naive Bayes seems to be a classification function. I don't think my problem is a classificaiton problem.  
  > **username_1**: If you already have some of these matches you can try and scale it to new data. i.e. your pk & fk cols would be x1, x2, match would be y1=1 no match y1=0. Do this for some data, then let it predict some data, update that data if the classification is wrong etc. ALT: you may want to look into ICA which can predict topics. https://www.cc.gatech.edu/~isbell/papers/isbell-ica-nips-1999.pdf  
  > **username_0**: thanks @username_1. We will need to test the bayes and research the ICA. That might be a good start.  
  
  ---
  
  ## 0 answers
  
  