Answers.Body: <p>Yes a variant of NLP processing could help find the correct number to extract and type of object in this data. </p>
  
  <p>Compared to the spreadsheet, the raw text data is ambiguous without understanding language to a reasonable depth, and without knowing the business context in order to extract the relevant information. </p>
  
  <p>For instance, you are expecting to extract "soup pan" and "covered pan", but not "capacity pot". Also that parts of phrases such as "30 mm" or "10-quart" are lower importance qualifiers, and not specifying quantity of something.</p>
  
  <p>The current state of the art for extracting this kind of data would be a <a href="https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks" rel="nofollow noreferrer">bidirectional LSTM (a type of Recurrent Neural Network)</a>. You would likely get it to flag the parts of each entry that were relevant to the tabulated data you wanted to extract, then feed those into a simpler stage that put them into the spreadsheet. However, there are two caveats:</p>
  
  <ul>
  <li><p>You need a lot of correctly-labelled training data to get reasonable performance. Using a word embedding layer, such as word2vec or GloVe, should significantly reduce the amount of training data required, but may require a careful pre-processing stage, and may be less useful when you have a lot of jargon in your data.</p></li>
  <li><p>Performance is never perfect, and the system can still make stupid mistakes, because it does not truly understand the text is is dealing with. That applies to all ML approaches to this problem, and likely also to coding up an "expert system", although it may be easier to write the expert system to recognise when it had failed and ask for help.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-18T07:16:16.697Z
Answers.LastActivityDate: 2018-04-18T07:24:03.137Z
Answers.LastEditDate: 2018-04-18T07:24:03.137Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Maybe a simple regex solve this question. But problably you need supervise, approve and disapprove anomalies.</p>
  
  <p>See this example, I had the same problem some time ago: <a href="https://stackoverflow.com/questions/50689935/regex-like-commands-python">https://stackoverflow.com/questions/50689935/regex-like-commands-python</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T19:07:45.250Z
Answers.LastActivityDate: 2018-06-21T19:07:45.250Z
Answers.UserDisplayName: GIA
Body: <p>I have order data, here's a sample:</p>
  
  <pre><code>Ninety-six (96) covered pans, desinated mark cutlery.
  5 vovered pans by knife co.
  (SEE SCHEDULE A FOR NUMBERS). 757 SOUP PANS
  115 10-quart capacity pots.
  Thirteen (13), 30 mm thick covered pans. 
  </code></pre>
  
  <p>I have over 50k rows of data such as this. In a perfect world, the above would need to be tabulated as such:</p>
  
  <pre><code>count, type
  96, covered pan
  5, covered pan
  757, soup pan
  115, pot
  13, covered pan
  </code></pre>
  
  <p>Could machine learning be the correct approach for a problem such as this?</p>
  
Tags: <machine-learning><ai-basics><datasets>
Title: Can machine learning help me digest asymmetrical order descriptions?
UserDisplayName: lewicki
ThreadText: # Can machine learning help me digest asymmetrical order descriptions?
  
  (Asked by: username_0 on 2018-04-18)
  
  I have order data, here's a sample:
  Ninety-six (96) covered pans, desinated mark cutlery.
  5 vovered pans by knife co.
  (SEE SCHEDULE A FOR NUMBERS). 757 SOUP PANS
  115 10-quart capacity pots.
  Thirteen (13), 30 mm thick covered pans. 
  
  I have over 50k rows of data such as this. In a perfect world, the above would need to be tabulated as such:
  count, type
  96, covered pan
  5, covered pan
  757, soup pan
  115, pot
  13, covered pan
  
  Could machine learning be the correct approach for a problem such as this?
  
  **Tags:** machine-learning, ai-basics, datasets
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-04-18. Score: 1
  
  Yes a variant of NLP processing could help find the correct number to extract and type of object in this data.
  
  Compared to the spreadsheet, the raw text data is ambiguous without understanding language to a reasonable depth, and without knowing the business context in order to extract the relevant information.
  
  For instance, you are expecting to extract "soup pan" and "covered pan", but not "capacity pot". Also that parts of phrases such as "30 mm" or "10-quart" are lower importance qualifiers, and not specifying quantity of something.
  
  The current state of the art for extracting this kind of data would be a [bidirectional LSTM (a type of Recurrent Neural Network)](https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks). You would likely get it to flag the parts of each entry that were relevant to the tabulated data you wanted to extract, then feed those into a simpler stage that put them into the spreadsheet. However, there are two caveats:
  
  * You need a lot of correctly-labelled training data to get reasonable performance. Using a word embedding layer, such as word2vec or GloVe, should significantly reduce the amount of training data required, but may require a careful pre-processing stage, and may be less useful when you have a lot of jargon in your data.
  * Performance is never perfect, and the system can still make stupid mistakes, because it does not truly understand the text is is dealing with. That applies to all ML approaches to this problem, and likely also to coding up an "expert system", although it may be easier to write the expert system to recognise when it had failed and ask for help.
  
  
  ---
  
  ### Answer by username_2 on 2018-06-21. Score: -1
  
  Maybe a simple regex solve this question. But problably you need supervise, approve and disapprove anomalies.
  
  See this example, I had the same problem some time ago: <https://stackoverflow.com/questions/50689935/regex-like-commands-python>
  
  
  ---
  
  