Answers.Body: <p>One of the renowned learning algorithms for NER tagging is the conditional random field (<a href="https://medium.com/data-science-in-your-pocket/named-entity-recognition-ner-using-conditional-random-fields-in-nlp-3660df22e95c" rel="nofollow noreferrer">CRF</a>). As you can see in the provided link, s<strong>equence labeling</strong> algorithms such as RNN with LSTM‌ can be used to named entity recognition as well. By the way, you can find an implementation of the CRF for NER tagging in <a href="https://www.depends-on-the-definition.com/named-entity-recognition-conditional-random-fields-python/" rel="nofollow noreferrer">this source</a>.</p>
  
  <p>Notice that, the method of providing training data can be helpful to pass the data into the standard libraries of CRF without any extra preprocessing. One of the standard methods is BIO method(B (Begin), I (Interior), and E (End)). You can find more about it in <a href="https://datascience.stackexchange.com/a/63400/27665">this post</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T19:37:06.440Z
Answers.Text: I think if you summarize the BIO method a bit, it can be a big lightbulb moment for understanding machine learning for language tasks. A tokenizer divides a text into a sequence of tokens, and the NER system labels each token as beginning, interior, or end of a name.
Answers.UserDisplayName: Jetpack
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T19:40:31.873Z
Answers.Text: @Jetpack I've refered it through a link. There is a summaried version of the BIO.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T10:11:55.397Z
Answers.LastActivityDate: 2020-06-15T10:21:03.363Z
Answers.LastEditDate: 2020-06-15T10:21:03.363Z
Answers.UserDisplayName: OmG
Answers.Body: <p>There are different algorithms, each with their advantages and disadvantages.</p>
  
  <ol>
  <li><p>Gazetteers: these have lists of the entities to be recognised, eg list of countries, cities, people, companies, whatever is required. They typically use a fuzzy matching algorithm to capture cases where the entity is not written in exactly the same way as in the list. For example, <em>USA</em> or <em>U.S.A.</em>, <em>United States</em>, <em>United States of America</em>, <em>US of A</em>, etc. Advantage: generally good precision, ie can identify known entities. Disadvantage: can only find known entities</p></li>
  <li><p>Contextual Clues: here you have patterns that you find in the text, eg <em>[PERSON], chairman of [COMPANY]</em>. In this case, sentences like <em>Jeff Bezos, chairman of Amazon,</em> will match, even if you have never come across either Bezos or Amazon. Advantage: can find entities you didn't know about. Disadvantages: could end up with false positives, might be quite labour-intensive to come up with patterns; patterns depend on the domain (newspapers vs textbooks vs novels etc_</p></li>
  <li><p>Structural description: this is basically a 'grammar' describing what your entities look like, eg (in some kind of pseudo-regex): <code>title? [first_name|initial] [middle_name|initial]? surname</code> would match "Mr J. R. Ewing" or "Bob Smith". Similar descriptions could match typical company names; you'd probably still need lists of possible surnames or firstnames. Advantages: some flexibility, and potentially good precision. Disadvantages: patterns need to be developed and maintained.</p></li>
  </ol>
  
  <p>Ideally you would want to combine all three approaches for a hybrid one to get the advantages of recognising unknown entities while keeping excess false positives in check.</p>
  
  <p>There might also be other machine-learning approaches, but I'm not too familiar with those. The main problem is that they are hard to fine-tune or work out why they do what they do.</p>
  
  <p><strong>UPDATE:</strong> A good starting point would be to use a gazetteer-based approach to annotate some training data, and use that to identify contextual patterns. You can then use that data to train a machine learning approach (see OmG's answer on CRF) to broaden the approach; and then you add newly recognised entities to your list.</p>
  
  <p>Ideally you would want to have a gazette as your main database to avoid false positives, and use machine-learning or contextual patterns to just capture previously unseen entities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T10:11:39.280Z
Answers.LastActivityDate: 2020-06-15T10:53:58.740Z
Answers.LastEditDate: 2020-06-15T10:53:58.740Z
Answers.UserDisplayName: Oliver Mason
Body: <blockquote>
    <p><strong><a href="https://www.kdnuggets.com/2018/08/named-entity-recognition-practitioners-guide-nlp-4.html" rel="nofollow noreferrer">Named entity recognition (NER)</a>, also known as entity chunking/extraction, is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.</strong></p>
  </blockquote>
  
  <p>Briefly, how does NER work? What are the main ideas behind it? And which algorithms are used to perform NER? </p>
  
Tags: <natural-language-processing><named-entity-recognition>
Title: What are the main ideas behind NER?
UserDisplayName: Pluviophile
ThreadText: # What are the main ideas behind NER?
  
  (Asked by: username_0 on 2020-06-15)
  
  > **[Named entity recognition (NER)](https://www.kdnuggets.com/2018/08/named-entity-recognition-practitioners-guide-nlp-4.html), also known as entity chunking/extraction, is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.**
  
  Briefly, how does NER work? What are the main ideas behind it? And which algorithms are used to perform NER?
  
  **Tags:** natural-language-processing, named-entity-recognition
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-06-15. Score: 3 | Accepted answer
  
  There are different algorithms, each with their advantages and disadvantages.
  
  1. Gazetteers: these have lists of the entities to be recognised, eg list of countries, cities, people, companies, whatever is required. They typically use a fuzzy matching algorithm to capture cases where the entity is not written in exactly the same way as in the list. For example, *USA* or *U.S.A.*, *United States*, *United States of America*, *US of A*, etc. Advantage: generally good precision, ie can identify known entities. Disadvantage: can only find known entities
  2. Contextual Clues: here you have patterns that you find in the text, eg *[PERSON], chairman of [COMPANY]*. In this case, sentences like *Jeff Bezos, chairman of Amazon,* will match, even if you have never come across either Bezos or Amazon. Advantage: can find entities you didn't know about. Disadvantages: could end up with false positives, might be quite labour-intensive to come up with patterns; patterns depend on the domain (newspapers vs textbooks vs novels etc\_
  3. Structural description: this is basically a 'grammar' describing what your entities look like, eg (in some kind of pseudo-regex): `title? [first_name|initial] [middle_name|initial]? surname` would match "Mr J. R. Ewing" or "Bob Smith". Similar descriptions could match typical company names; you'd probably still need lists of possible surnames or firstnames. Advantages: some flexibility, and potentially good precision. Disadvantages: patterns need to be developed and maintained.
  
  Ideally you would want to combine all three approaches for a hybrid one to get the advantages of recognising unknown entities while keeping excess false positives in check.
  
  There might also be other machine-learning approaches, but I'm not too familiar with those. The main problem is that they are hard to fine-tune or work out why they do what they do.
  
  **UPDATE:** A good starting point would be to use a gazetteer-based approach to annotate some training data, and use that to identify contextual patterns. You can then use that data to train a machine learning approach (see OmG's answer on CRF) to broaden the approach; and then you add newly recognised entities to your list.
  
  Ideally you would want to have a gazette as your main database to avoid false positives, and use machine-learning or contextual patterns to just capture previously unseen entities.
  
  
  ---
  
  ### Answer by username_1 on 2020-06-15. Score: 3
  
  One of the renowned learning algorithms for NER tagging is the conditional random field ([CRF](https://medium.com/data-science-in-your-pocket/named-entity-recognition-ner-using-conditional-random-fields-in-nlp-3660df22e95c)). As you can see in the provided link, s**equence labeling** algorithms such as RNN with LSTM‌ can be used to named entity recognition as well. By the way, you can find an implementation of the CRF for NER tagging in [this source](https://www.depends-on-the-definition.com/named-entity-recognition-conditional-random-fields-python/).
  
  Notice that, the method of providing training data can be helpful to pass the data into the standard libraries of CRF without any extra preprocessing. One of the standard methods is BIO method(B (Begin), I (Interior), and E (End)). You can find more about it in [this post](https://datascience.stackexchange.com/a/63400/27665).
  
  > **Comments:**
  > **username_2**: I think if you summarize the BIO method a bit, it can be a big lightbulb moment for understanding machine learning for language tasks. A tokenizer divides a text into a sequence of tokens, and the NER system labels each token as beginning, interior, or end of a name.  
  > **username_1**: @Jetpack I've refered it through a link. There is a summaried version of the BIO.  
  
  ---
  
  