<p>I'm new to AI/ML and I want to research and learn about techniques that could help me to solve this complex task. Any hint would be appreciated.</p>
<p>Let me explain it with an example:</p>
<p>Let's look at two columns <code>PUR.SUPPLY.MTL_REQ_HDR_ID</code> and <code>MTL.PO_REQUISITION_HEADERS_TAB.ID</code>. It is likely they are related (one is the FK and the other one is PK).</p>
<p><a href="https://i.sstatic.net/etgPH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/etgPH.png" alt="enter image description here" /></a></p>
<p>As a human I was able to do it by doing the following:</p>
<ol>
<li>I decoded abbreviations,</li>
<li>I identified context (MTL module),</li>
<li>I identified the subject (requisition header),</li>
<li>I identified ID keyword,</li>
<li>I identified irrelevant information (TAB postfix),</li>
<li>I matched words that are not in the exact order,</li>
<li>I estimated which elements/words do not have to match/occur,</li>
</ol>
<p>I would like to match millions of columns relatively quickly (seconds). I would like algorithm to learn:</p>
<ol>
<li>what words are likely context,</li>
<li>what are irrelevant,</li>
<li>what are subjects,</li>
<li>ideally learn some patterns (prefixes, postfixes, name formats, etc),</li>
<li>based on user responses - approve/reject match,</li>
<li>build dictionary of abbreviations,</li>
<li>estimate probability...</li>
</ol>
<p>I know this is a complex task, but maybe you know a technique, tool, library, article, example ... anything that could be helpful? Any help would be appreciated.</p>
<p>Thanks.</p>

CC BY-SA 4.0
2021-06-13T04:06:30.350Z
In general Bayesian networks/Naive Bayes do well at text processing. As a library sci-kit has an implementation.
A_Arnold
CC BY-SA 4.0
2021-06-13T18:42:35.857Z
Thanks @A_Arnold, Naive Bayes seems to be a classification function. I don't think my problem is a classificaiton problem.
Bad Pitt
CC BY-SA 4.0
2021-06-13T20:23:02.593Z
If you already have some of these matches you can try and scale it to new data. i.e. your pk & fk cols would be x1, x2, match would be y1=1 no match y1=0. Do this for some data, then let it predict some data, update that data if the classification is wrong etc. ALT: you may want to look into ICA which can predict topics. https://www.cc.gatech.edu/~isbell/papers/isbell-ica-nips-1999.pdf
A_Arnold
CC BY-SA 4.0
2021-06-14T21:34:47.640Z
thanks @A_Arnold. We will need to test the bayes and research the ICA. That might be a good start.
Bad Pitt
<machine-learning><natural-language-understanding>
Text matching: fuzzy names matching with learning
Bad Pitt
# Text matching: fuzzy names matching with learning

(Asked by: username_0 on 2021-06-12)

I'm new to AI/ML and I want to research and learn about techniques that could help me to solve this complex task. Any hint would be appreciated.

Let me explain it with an example:

Let's look at two columns `PUR.SUPPLY.MTL_REQ_HDR_ID` and `MTL.PO_REQUISITION_HEADERS_TAB.ID`. It is likely they are related (one is the FK and the other one is PK).

[![enter image description here](https://i.sstatic.net/etgPH.png)](https://i.sstatic.net/etgPH.png)

As a human I was able to do it by doing the following:

1. I decoded abbreviations,
2. I identified context (MTL module),
3. I identified the subject (requisition header),
4. I identified ID keyword,
5. I identified irrelevant information (TAB postfix),
6. I matched words that are not in the exact order,
7. I estimated which elements/words do not have to match/occur,

I would like to match millions of columns relatively quickly (seconds). I would like algorithm to learn:

1. what words are likely context,
2. what are irrelevant,
3. what are subjects,
4. ideally learn some patterns (prefixes, postfixes, name formats, etc),
5. based on user responses - approve/reject match,
6. build dictionary of abbreviations,
7. estimate probability...

I know this is a complex task, but maybe you know a technique, tool, library, article, example ... anything that could be helpful? Any help would be appreciated.

Thanks.

**Tags:** machine-learning, natural-language-understanding

> **Comments:**
> **username_1**: In general Bayesian networks/Naive Bayes do well at text processing. As a library sci-kit has an implementation.  
> **username_0**: Thanks @username_1, Naive Bayes seems to be a classification function. I don't think my problem is a classificaiton problem.  
> **username_1**: If you already have some of these matches you can try and scale it to new data. i.e. your pk & fk cols would be x1, x2, match would be y1=1 no match y1=0. Do this for some data, then let it predict some data, update that data if the classification is wrong etc. ALT: you may want to look into ICA which can predict topics. https://www.cc.gatech.edu/~isbell/papers/isbell-ica-nips-1999.pdf  
> **username_0**: thanks @username_1. We will need to test the bayes and research the ICA. That might be a good start.  

---

## 0 answers

