Answers.Body: <p>You should first segregate the rejected samples. You can use then use string matching or something more complex (like creating embeddings and then, taking L2 distance between them) between the different field names you have and the comment for rejection. Whichever field gets the highest score, you increase the rejection count for that field. In the end, you will have a tally of who is your biggest enemy.</p>
  <p>You can create some rules which prevent injection of wrong data (like your password should be 7 characters long or something along these lines) or post-process your entries to match a uniform format.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T22:00:44.800Z
Answers.Text: Okay, I believe I understand. Are you familiar with an L2 library or functions in Python by any chance?
  
  Also, I tried using correlation coefficients to find which fields correlated most highly with the outcome. Do you have an opinion on that option, albeit less sophisticated than an actual algorithm?
Answers.UserDisplayName: Petar Luketina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T22:07:18.653Z
Answers.Text: L2 distance you can easily find using numpy's linalg.norm or in scipy.spatial.distance.euclidean. But, you need to convert textual data to numbers. Fields which have negative correlation, if the outcome goes, they down, then they have an antagonistic relationship. Just use string matching and find the bad variables that is good starting approach. Afterwards, you need to devise some rules or methods things, that will take more time.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T19:37:46.327Z
Answers.LastActivityDate: 2021-02-25T19:37:46.327Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I am working with tabular data that is similar to the below:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Name</th>
  <th>Phone Number</th>
  <th>ISO3 Country</th>
  <th>Amount</th>
  <th>Email</th>
  <th>...</th>
  <th>...</th>
  <th>Outcome</th>
  <th>Possible Reason</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Leona Sunfurry</td>
  <td>(555)-555-5555</td>
  <td>United States</td>
  <td>58.96</td>
  <td>leo_sun@gmail.com</td>
  <td>...</td>
  <td>...</td>
  <td>0</td>
  <td>Not ISO3 country</td>
  </tr>
  <tr>
  <td>Diana Moonglory</td>
  <td>(333)-555-5555</td>
  <td>USA</td>
  <td>8.32</td>
  <td>di.moon@gmail.com</td>
  <td>...</td>
  <td>...</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>Fiora Quik</td>
  <td>(111)-555-5555</td>
  <td>FRA</td>
  <td>0.35</td>
  <td>null</td>
  <td>...</td>
  <td>...</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>Darius Guy</td>
  <td>12345678901234</td>
  <td>CAN</td>
  <td>555.01</td>
  <td>null</td>
  <td>...</td>
  <td>...</td>
  <td>0</td>
  <td>Too many digits in phone</td>
  </tr>
  <tr>
  <td>LULU</td>
  <td>(333)-555-5555</td>
  <td>CAN</td>
  <td>0.00</td>
  <td>null</td>
  <td>...</td>
  <td>...</td>
  <td>0</td>
  <td>Odd name format</td>
  </tr>
  <tr>
  <td>Eve K.</td>
  <td>(111)-555-5555</td>
  <td>FRA</td>
  <td>69.25</td>
  <td>e.k@gmail.com</td>
  <td>...</td>
  <td>...</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>Lucian Light</td>
  <td>(999)-555-5555</td>
  <td>ENG</td>
  <td>65.00</td>
  <td>null</td>
  <td>...</td>
  <td>...</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>Lux D.</td>
  <td>(333)-555-5555</td>
  <td>USA</td>
  <td>11.64</td>
  <td>test@test.com</td>
  <td>...</td>
  <td>...</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>Jarvin Crown</td>
  <td>(333)-555-5555</td>
  <td>USA</td>
  <td>1357.13</td>
  <td>j4@gmail.com</td>
  <td>...</td>
  <td>...</td>
  <td>0</td>
  <td>Unknown reason</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>The table contains information about users. Some of the fields are user-generated while others are generated by the program (like device location, amount, etc.). When this data is collected, it is sent to third parties (we will say a bank). Sometimes the bank rejects the data and it is not good for our users. The rejection could have happened because the user did not input the data correctly or the banks did not like how a field is formatted despite the data being correct and acceptable to other banks.</p>
  <p>So we want to find the fields that are causing the most errors and how to fix the issue.</p>
  <p>Does it make sense to do pattern recognition on the values to find the reason why the row was rejected? It would need to be an alpha-numeric type of algorithm, it seems.</p>
  <p>We know the outcomes from the bank which is labeled as <code>Outcome</code>.  Although we have labeled data, it still feels like we need an unsupervised learning algorithm because we do not have labels on <em>why</em> the rows of data were rejected.</p>
  <p>Does anyone know what type of algorithm would be best? Any feedback would be appreciated!</p>
  
Tags: <unsupervised-learning><pattern-recognition><text-classification><algorithm-request>
Title: Which algorithm can be used for extracting text patterns in tabular data?
UserDisplayName: Petar Luketina
ThreadText: # Which algorithm can be used for extracting text patterns in tabular data?
  
  (Asked by: username_0 on 2021-02-25)
  
  I am working with tabular data that is similar to the below:
  
  | Name | Phone Number | ISO3 Country | Amount | Email | ... | ... | Outcome | Possible Reason |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
  | Leona Sunfurry | (555)-555-5555 | United States | 58.96 | leo\_sun@gmail.com | ... | ... | 0 | Not ISO3 country |
  | Diana Moonglory | (333)-555-5555 | USA | 8.32 | di.moon@gmail.com | ... | ... | 1 |  |
  | Fiora Quik | (111)-555-5555 | FRA | 0.35 | null | ... | ... | 1 |  |
  | Darius Guy | 12345678901234 | CAN | 555.01 | null | ... | ... | 0 | Too many digits in phone |
  | LULU | (333)-555-5555 | CAN | 0.00 | null | ... | ... | 0 | Odd name format |
  | Eve K. | (111)-555-5555 | FRA | 69.25 | e.k@gmail.com | ... | ... | 1 |  |
  | Lucian Light | (999)-555-5555 | ENG | 65.00 | null | ... | ... | 1 |  |
  | Lux D. | (333)-555-5555 | USA | 11.64 | test@test.com | ... | ... | 1 |  |
  | Jarvin Crown | (333)-555-5555 | USA | 1357.13 | j4@gmail.com | ... | ... | 0 | Unknown reason |
  
  The table contains information about users. Some of the fields are user-generated while others are generated by the program (like device location, amount, etc.). When this data is collected, it is sent to third parties (we will say a bank). Sometimes the bank rejects the data and it is not good for our users. The rejection could have happened because the user did not input the data correctly or the banks did not like how a field is formatted despite the data being correct and acceptable to other banks.
  
  So we want to find the fields that are causing the most errors and how to fix the issue.
  
  Does it make sense to do pattern recognition on the values to find the reason why the row was rejected? It would need to be an alpha-numeric type of algorithm, it seems.
  
  We know the outcomes from the bank which is labeled as `Outcome`. Although we have labeled data, it still feels like we need an unsupervised learning algorithm because we do not have labels on *why* the rows of data were rejected.
  
  Does anyone know what type of algorithm would be best? Any feedback would be appreciated!
  
  **Tags:** unsupervised-learning, pattern-recognition, text-classification, algorithm-request
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-25. Score: 1 | Accepted answer
  
  You should first segregate the rejected samples. You can use then use string matching or something more complex (like creating embeddings and then, taking L2 distance between them) between the different field names you have and the comment for rejection. Whichever field gets the highest score, you increase the rejection count for that field. In the end, you will have a tally of who is your biggest enemy.
  
  You can create some rules which prevent injection of wrong data (like your password should be 7 characters long or something along these lines) or post-process your entries to match a uniform format.
  
  > **Comments:**
  > **username_0**: Okay, I believe I understand. Are you familiar with an L2 library or functions in Python by any chance?  Also, I tried using correlation coefficients to find which fields correlated most highly with the outcome. Do you have an opinion on that option, albeit less sophisticated than an actual algorithm?  
  > **username_1**: L2 distance you can easily find using numpy's linalg.norm or in scipy.spatial.distance.euclidean. But, you need to convert textual data to numbers. Fields which have negative correlation, if the outcome goes, they down, then they have an antagonistic relationship. Just use string matching and find the bad variables that is good starting approach. Afterwards, you need to devise some rules or methods things, that will take more time.  
  
  ---
  
  