Answers.Body: <p>It's certainly possible to treat this as a natural language processing problem, basically you're looking to assign "salience" scores to the text.</p>
  
  <p>Really, though, that's overkill for this kind of problem. Writing a regex or a CFG parser (or better: finding an existing parser) is likely to be easier and more reliable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:16:20.030Z
Answers.Text: Regex are out of the question. This problem cannot be solved by regex since the inputs are public and I can't write one for every mail software out there. The Talon library "sort of" does this, but the regex only "sort of" works for english.
Answers.UserDisplayName: hjf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:21:35.223Z
Answers.Text: I suspect you'd still find something like a CFG parser (e.g. made with YACC) more reliable and easier. While there are a lot of mail clients, you'd probably get 80%+ of traffic by covering the biggest 3 of them (outlook, gmail, thunderbird?). If you really can't use one, then "salience" scores are probably your best bet. There are some good python libraries for this I think: http://www.nltk.org/
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:27:25.937Z
Answers.Text: Talon actually does this. It claims that 90% of the time the "simple" approach works, and for the remaining 10% they used scikit-learn to train a model with the ENRON emails. I've tested the library. It  works perfectly with some emails, but fails miserably with others. The "others" are malformed HTML emails (companies like to slap their stupid, useless legalese right at the server to every outgoing email, breaking HTML mail)
Answers.UserDisplayName: hjf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:31:40.070Z
Answers.Text: It sounds like your concern is with very esoteric formats. The reason Talon's approach doesn't work for those is that they're 'out-of-sample': the ENRON dataset is just internal emails from ENRON, and won't include exotic cases. If you want to build something similar that does work for those cases, you'll need to gather many examples of the cases you want to cover, and then build your own model to cover the extra 10%. I suspect you'd still find it easier to do a CFG though... you'll likely need fewer examples for that, and the structured nature of HTML gives you a big leg up.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:40:53.873Z
Answers.Text: Let's change the focus a bit. The problem not really the extraction of text from emails, but rather how to classify the emails. I've told management maybe the way is not to try and use the current neural network (trained with text-only messages from twitter and facebook), but instead train another one with the full body of emails. Let the neural network learn to ignore HTML tags and legalese. We have about 1000 incoming emails every day so it's easy to get large datasets.
Answers.UserDisplayName: hjf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T19:41:21.213Z
Answers.Text: Ah! This clarifies things a lot. I think you should actually just use the raw text. Don't bother filtering out the unneeded HTML tags. If those turn out not to be useful, the network will learn to ignore them. If they turn out to be useful, the network will use them. In fact, this is the approach used by many email classification systems, not just neural nets (see, for instance, DMC and other compression-based approaches).
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T19:53:41.437Z
Answers.Text: The only other thing i'm worried about are long reply streams. Is there a way to tell the network that the deeper some text is into a string, the less relevant it may be?. I still think some sort of filtering will be needing. I spent most of today looking for patterns. gmail is nice enough to use blockquote and other gmail_* classes that make it easy to detect replies ("gmail_quote").
Answers.UserDisplayName: hjf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T19:59:59.287Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/81238/discussion-between-john-doucette-and-hjf).
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-09T22:13:37.827Z
Answers.Text: As a follow up: I spent the last 2 days writing regex to catch most of the signature and reply blocks. I found about 20 patterns in the sample of 1000 emails I took. while it's not perfect, it seems to be "good enough". the worst offender is, of course, Outlook, which seems to divide the replies with a div (because <hr> isn't good enough for microsoft). and this div depends on the user's language. Sometimes it specifies a height of 0cm, other times 0in... unbelievable.
Answers.UserDisplayName: hjf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-03T21:31:40.857Z
Answers.LastActivityDate: 2018-08-03T21:31:40.857Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>It is a surmountable problem for someone experienced in software architecture and machine learning.</p>
  
  <ol>
  <li>Render the message to a virtual display such as xvfb, headless Chrome, or phantomjs.</li>
  <li>Capture the text with selenium, watir, or some other DOM controller, addressing your HTML and DHTML complexity concern.</li>
  <li>OCR the text in inline images and insert it appropriately.</li>
  <li>Once you have text with only word, line, list item, and paragraph breaks as structural separators, you have adequate separation of style and language content to then use naive Bayesian or one of the more recent forms of unsupervised categorization to find the separation point between the body and the signature block.</li>
  </ol>
  
  <p>Extending your line of thinking, you may even be able to engineer a generative strategy for automated reply, but beware, this last feat is a dozen orders of magnitude more difficult than extracting text from HTML, DHTML, and typeset images and machine learning the separating signature blocks.</p>
  
  <p>This last feat, if done poorly, would get you in trouble with many of your email reply recipients, and, if done well, would place you ahead of Amazon, Apple, and Google.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:19:56.363Z
Answers.Text: Intriguing. I was actually wondering how to deal with the broken HTML usually found in emails. The second part you mention is irrelevant for us since the service we sell is actually humans to respond. But we need help categorizing the inputs and try and deliver to the right person. We're already doing this with tweets and facebook messages which are trivial being text-only. But email is a different beast.
Answers.UserDisplayName: hjf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-05T05:19:44.967Z
Answers.LastActivityDate: 2018-08-05T05:19:44.967Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I need to retrieve just the text from emails. The emails can be in HTML format, and can contain huge signatures, disclaimer legalese, and broken HTML from dozens of forwards and replies. But, I only want the actual email message and not any other cruft such as the whole quotation block, signatures, etc.</p>
  
  <p>This isn't really a problem that could be solved with regex because HTML mail can get very, VERY messy.</p>
  
  <p>Could a neural network perform this task? What kind of problem is this? Classification? Feature selection?</p>
  
Tags: <neural-networks><classification>
Title: What kind of problem is "email text extraction"?
UserDisplayName: hjf
ThreadText: # What kind of problem is "email text extraction"?
  
  (Asked by: username_0 on 2018-08-03)
  
  I need to retrieve just the text from emails. The emails can be in HTML format, and can contain huge signatures, disclaimer legalese, and broken HTML from dozens of forwards and replies. But, I only want the actual email message and not any other cruft such as the whole quotation block, signatures, etc.
  
  This isn't really a problem that could be solved with regex because HTML mail can get very, VERY messy.
  
  Could a neural network perform this task? What kind of problem is this? Classification? Feature selection?
  
  **Tags:** neural-networks, classification
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-03. Score: 2
  
  It's certainly possible to treat this as a natural language processing problem, basically you're looking to assign "salience" scores to the text.
  
  Really, though, that's overkill for this kind of problem. Writing a regex or a CFG parser (or better: finding an existing parser) is likely to be easier and more reliable.
  
  > **Comments:**
  > **username_0**: Regex are out of the question. This problem cannot be solved by regex since the inputs are public and I can't write one for every mail software out there. The Talon library "sort of" does this, but the regex only "sort of" works for english.  
  > **username_1**: I suspect you'd still find something like a CFG parser (e.g. made with YACC) more reliable and easier. While there are a lot of mail clients, you'd probably get 80%+ of traffic by covering the biggest 3 of them (outlook, gmail, thunderbird?). If you really can't use one, then "salience" scores are probably your best bet. There are some good python libraries for this I think: http://www.nltk.org/  
  > **username_0**: Talon actually does this. It claims that 90% of the time the "simple" approach works, and for the remaining 10% they used scikit-learn to train a model with the ENRON emails. I've tested the library. It  works perfectly with some emails, but fails miserably with others. The "others" are malformed HTML emails (companies like to slap their stupid, useless legalese right at the server to every outgoing email, breaking HTML mail)  
  > **username_1**: It sounds like your concern is with very esoteric formats. The reason Talon's approach doesn't work for those is that they're 'out-of-sample': the ENRON dataset is just internal emails from ENRON, and won't include exotic cases. If you want to build something similar that does work for those cases, you'll need to gather many examples of the cases you want to cover, and then build your own model to cover the extra 10%. I suspect you'd still find it easier to do a CFG though... you'll likely need fewer examples for that, and the structured nature of HTML gives you a big leg up.  
  > **username_0**: Let's change the focus a bit. The problem not really the extraction of text from emails, but rather how to classify the emails. I've told management maybe the way is not to try and use the current neural network (trained with text-only messages from twitter and facebook), but instead train another one with the full body of emails. Let the neural network learn to ignore HTML tags and legalese. We have about 1000 incoming emails every day so it's easy to get large datasets.  
  > **username_1**: Ah! This clarifies things a lot. I think you should actually just use the raw text. Don't bother filtering out the unneeded HTML tags. If those turn out not to be useful, the network will learn to ignore them. If they turn out to be useful, the network will use them. In fact, this is the approach used by many email classification systems, not just neural nets (see, for instance, DMC and other compression-based approaches).  
  > **username_0**: The only other thing i'm worried about are long reply streams. Is there a way to tell the network that the deeper some text is into a string, the less relevant it may be?. I still think some sort of filtering will be needing. I spent most of today looking for patterns. gmail is nice enough to use blockquote and other gmail_* classes that make it easy to detect replies ("gmail_quote").  
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/81238/discussion-between-username_1-username_1-and-hjf).  
  > **username_0**: As a follow up: I spent the last 2 days writing regex to catch most of the signature and reply blocks. I found about 20 patterns in the sample of 1000 emails I took. while it's not perfect, it seems to be "good enough". the worst offender is, of course, Outlook, which seems to divide the replies with a div (because <hr> isn't good enough for microsoft). and this div depends on the user's language. Sometimes it specifies a height of 0cm, other times 0in... unbelievable.  
  
  ---
  
  ### Answer by username_2 on 2018-08-05. Score: 1
  
  It is a surmountable problem for someone experienced in software architecture and machine learning.
  
  1. Render the message to a virtual display such as xvfb, headless Chrome, or phantomjs.
  2. Capture the text with selenium, watir, or some other DOM controller, addressing your HTML and DHTML complexity concern.
  3. OCR the text in inline images and insert it appropriately.
  4. Once you have text with only word, line, list item, and paragraph breaks as structural separators, you have adequate separation of style and language content to then use naive Bayesian or one of the more recent forms of unsupervised categorization to find the separation point between the body and the signature block.
  
  Extending your line of thinking, you may even be able to engineer a generative strategy for automated reply, but beware, this last feat is a dozen orders of magnitude more difficult than extracting text from HTML, DHTML, and typeset images and machine learning the separating signature blocks.
  
  This last feat, if done poorly, would get you in trouble with many of your email reply recipients, and, if done well, would place you ahead of Amazon, Apple, and Google.
  
  > **Comments:**
  > **username_0**: Intriguing. I was actually wondering how to deal with the broken HTML usually found in emails. The second part you mention is irrelevant for us since the service we sell is actually humans to respond. But we need help categorizing the inputs and try and deliver to the right person. We're already doing this with tweets and facebook messages which are trivial being text-only. But email is a different beast.  
  
  ---
  
  