Answers.Body: <p>It depends how you recognise the entities.</p>
  
  <p>If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with.</p>
  
  <p>However, if you use contextual rules, then stop words might be vital to identify certain contexts, so by removing stop words you lose information about the entity's environment. For example, if <strong>[work] at {organisation}</strong> is a rule you use to identify companies etc, then this wouldn't work if you take out the <em>at</em>.</p>
  
  <p>You will also have problems if the stop words are part of an entity, eg the town <em>Stoke-on-Trent</em>. If hyphens are used to split tokens, then you won't be able to recognise it if the <em>on</em> is discarded.</p>
  
  <p>In general I think stop words should mostly be kept; apart from information retrieval where they are not all that useful in an inverted index you will always lose something. If stop words were really pointless, then they would have been discarded from languages a long time ago. In the old days they were a useful way of reducing the demand on computational resources without losing too much, but nowadays I would say this is not really a problem anymore.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-04T16:15:05.423Z
Answers.LastActivityDate: 2020-03-04T16:15:05.423Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Removing stop words can significantly speed up named entity recognition (NER) modeling by reducing the number of tokens in a document.</p>
  
  <p>Are stop words critical to get correct NER performance?</p>
  
Tags: <natural-language-processing><data-preprocessing>
Title: Is it recommended to remove stop words before named entity recognition?
UserDisplayName: nous
ThreadText: # Is it recommended to remove stop words before named entity recognition?
  
  (Asked by: username_0 on 2020-03-04)
  
  Removing stop words can significantly speed up named entity recognition (NER) modeling by reducing the number of tokens in a document.
  
  Are stop words critical to get correct NER performance?
  
  **Tags:** natural-language-processing, data-preprocessing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-04. Score: 2
  
  It depends how you recognise the entities.
  
  If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with.
  
  However, if you use contextual rules, then stop words might be vital to identify certain contexts, so by removing stop words you lose information about the entity's environment. For example, if **[work] at {organisation}** is a rule you use to identify companies etc, then this wouldn't work if you take out the *at*.
  
  You will also have problems if the stop words are part of an entity, eg the town *Stoke-on-Trent*. If hyphens are used to split tokens, then you won't be able to recognise it if the *on* is discarded.
  
  In general I think stop words should mostly be kept; apart from information retrieval where they are not all that useful in an inverted index you will always lose something. If stop words were really pointless, then they would have been discarded from languages a long time ago. In the old days they were a useful way of reducing the demand on computational resources without losing too much, but nowadays I would say this is not really a problem anymore.
  
  
  ---
  
  