Answers.Body: <p>I don't know that NER is the right approach here. It seems to me that you want to find words for certain technical components in free texts, written in German. But &quot;Festplatte&quot; is not a named entity. Named entities (cities, companies, countries, etc) in English (and other languages) are usually capitalised, so relatively easy to spot. In German this won't work as <em>every</em> noun is capitalised, named entity or not. But even in English a NER wouldn't help you with &quot;hard disk&quot;, as it's not what is commonly understood as a named entity.</p>
  <p>It's not really an AI solution, but I would get a list of relevant components (eg from a dictionary), and then simply match those in texts. Instead of annotating existing texts, you simply add the words to a list, which would be a bit quicker, generally. And list lookup is very easy to implement.</p>
  <p>This, I think, would work a lot better than a machine learning approach. If you find that your technicians often mis-spell the words, use a fuzzy matching algorithm such as <a href="https://en.wikipedia.org/wiki/Levenshtein_distance" rel="nofollow noreferrer">Levenshtein distance</a> to allow for close matches; this might also help with inflections.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-17T13:39:42.930Z
Answers.LastActivityDate: 2021-06-17T13:39:42.930Z
Answers.UserDisplayName: Oliver Mason
Body: <p>I'm starting a project where I want to extract keywords from given messages. The keywords are for example something like: &quot;hard disk&quot;, &quot;watch&quot; or other technical components. I'm working with a dataset where a technician wrote a small text if he maintenanced something on a given object.</p>
  <p>The messages are often very different in their form. For example sometimes the messages start with the repaired object and sometimes with the current date.</p>
  <p>I looked into some NER-Libaries and it doesn't seem like they can handle tasks like that. Especially the German language makes it hard for those libaries to detect entities.</p>
  <p>I had the idea to use CRFsuite to train my own NER-model. But I'm not sure how accurate the outcome will be. It would mean that I have to tag <em>a lot</em> of training data and I'm not sure if the outcome will match the time I have to spend to tag those keywords.</p>
  <p>Does anybody have any experience with such custom NER-models? How accurate can such a model extract wanted keywords?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T23:55:33.193Z
Comments.Text: Hello. [Could you please put your main **specific** question in the title?](https://ai.meta.stackexchange.com/q/1654/2444)
Comments.UserDisplayName: nbro
Tags: <machine-learning><natural-language-processing><supervised-learning><named-entity-recognition>
Title: Extracting keywords from messages
UserDisplayName: gurke
ThreadText: # Extracting keywords from messages
  
  (Asked by: username_0 on 2021-06-17)
  
  I'm starting a project where I want to extract keywords from given messages. The keywords are for example something like: "hard disk", "watch" or other technical components. I'm working with a dataset where a technician wrote a small text if he maintenanced something on a given object.
  
  The messages are often very different in their form. For example sometimes the messages start with the repaired object and sometimes with the current date.
  
  I looked into some NER-Libaries and it doesn't seem like they can handle tasks like that. Especially the German language makes it hard for those libaries to detect entities.
  
  I had the idea to use CRFsuite to train my own NER-model. But I'm not sure how accurate the outcome will be. It would mean that I have to tag *a lot* of training data and I'm not sure if the outcome will match the time I have to spend to tag those keywords.
  
  Does anybody have any experience with such custom NER-models? How accurate can such a model extract wanted keywords?
  
  **Tags:** machine-learning, natural-language-processing, supervised-learning, named-entity-recognition
  > **Comments:**
  > **username_1**: Hello. [Could you please put your main **specific** question in the title?](https://ai.meta.stackexchange.com/q/1654/2444)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-06-17. Score: 0
  
  I don't know that NER is the right approach here. It seems to me that you want to find words for certain technical components in free texts, written in German. But "Festplatte" is not a named entity. Named entities (cities, companies, countries, etc) in English (and other languages) are usually capitalised, so relatively easy to spot. In German this won't work as *every* noun is capitalised, named entity or not. But even in English a NER wouldn't help you with "hard disk", as it's not what is commonly understood as a named entity.
  
  It's not really an AI solution, but I would get a list of relevant components (eg from a dictionary), and then simply match those in texts. Instead of annotating existing texts, you simply add the words to a list, which would be a bit quicker, generally. And list lookup is very easy to implement.
  
  This, I think, would work a lot better than a machine learning approach. If you find that your technicians often mis-spell the words, use a fuzzy matching algorithm such as [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) to allow for close matches; this might also help with inflections.
  
  
  ---
  
  