Answers.Body: <p>One of the renowned methods is using the TF-IDF method. You can define a lower bound for the TF-IDF score of the bigram or trigram to identify it as a keyword. <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf" rel="nofollow noreferrer">Here</a>, you can find more information about it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T22:32:25.987Z
Answers.Text: Isn't tf-idf only used to rank words in a document with respect to all others in that document and in that database? How can you use tf-idf to actually find "concepts"? It's true that the OP did not define "concepts" though, but only gave an example. Maybe tf-idf is sufficient, but I think it's better if we ask for clarification about what a concept is.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T13:56:02.790Z
Answers.Text: @nbro As I know here the concept means a meaningful bigram or trigram that can be happened in the meaningful number of times. Hence, TF-IDF can be a good score to find these meaningful concepts. I agree that TF can be meaningful measure for bigrams or trigrams as well.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T16:46:03.170Z
Answers.LastActivityDate: 2020-07-07T16:46:03.170Z
Answers.UserDisplayName: OmG
Body: <p>I have many text documents and I want to identify concepts in these documents in an unsupervised manner. One of my problems is that the concepts can be bigrams, trigrams, or even longer.</p>
  <p>So, for example, out of all the bigrams, how can I identify the ones that are more likely to represent a concept?</p>
  <p>A concept could be &quot;machine learning&quot;.</p>
  <p>Are you aware of any standard approaches to solve this problem?</p>
  <p><strong>Edit:</strong> The corpus I am working with consists of papers accessed from web of science. That is, they are all in some given domain niche. I want to extract words, bigrams, trigrams... that represent common concepts/buzzwords from these papers. These could be &quot;Automated machine learning&quot;, &quot;natural language processing&quot; et cetera. I need to be able to distinguish these from other common n-grams such as &quot;New York&quot;, &quot;Barack Obama&quot;,...</p>
  <p>I know that I could do this using a NER approach but this would require hand-labelling. Are you aware of any unsupervised ways to approach this problem? Or even a semi-superised method with little labelled data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-07T22:35:35.800Z
Comments.Text: I think it's important that you define what you mean by "concept", even though you provide an example. Is it a noun or a name? Are all nouns concepts? And I understood that concepts can be composed of more than one word. I think you want to exclude adjectives, adverbs, and verbs, but you should clarify this.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-08T11:39:03.183Z
Comments.Text: @nbro, thank you for the comment. I added a clarification that hopefully makes it clearer.
Comments.UserDisplayName: Haffi112
Tags: <natural-language-processing>
Title: How can I identify bigrams and trigrams that represent concepts?
UserDisplayName: Haffi112
ThreadText: # How can I identify bigrams and trigrams that represent concepts?
  
  (Asked by: username_0 on 2020-07-07)
  
  I have many text documents and I want to identify concepts in these documents in an unsupervised manner. One of my problems is that the concepts can be bigrams, trigrams, or even longer.
  
  So, for example, out of all the bigrams, how can I identify the ones that are more likely to represent a concept?
  
  A concept could be "machine learning".
  
  Are you aware of any standard approaches to solve this problem?
  
  **Edit:** The corpus I am working with consists of papers accessed from web of science. That is, they are all in some given domain niche. I want to extract words, bigrams, trigrams... that represent common concepts/buzzwords from these papers. These could be "Automated machine learning", "natural language processing" et cetera. I need to be able to distinguish these from other common n-grams such as "New York", "Barack Obama",...
  
  I know that I could do this using a NER approach but this would require hand-labelling. Are you aware of any unsupervised ways to approach this problem? Or even a semi-superised method with little labelled data?
  
  **Tags:** natural-language-processing
  
  > **Comments:**
  > **username_1**: I think it's important that you define what you mean by "concept", even though you provide an example. Is it a noun or a name? Are all nouns concepts? And I understood that concepts can be composed of more than one word. I think you want to exclude adjectives, adverbs, and verbs, but you should clarify this.  
  > **username_0**: @username_1, thank you for the comment. I added a clarification that hopefully makes it clearer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-07. Score: -1
  
  One of the renowned methods is using the TF-IDF method. You can define a lower bound for the TF-IDF score of the bigram or trigram to identify it as a keyword. [Here](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), you can find more information about it.
  
  > **Comments:**
  > **username_1**: Isn't tf-idf only used to rank words in a document with respect to all others in that document and in that database? How can you use tf-idf to actually find "concepts"? It's true that the OP did not define "concepts" though, but only gave an example. Maybe tf-idf is sufficient, but I think it's better if we ask for clarification about what a concept is.  
  > **username_2**: @username_1 As I know here the concept means a meaningful bigram or trigram that can be happened in the meaningful number of times. Hence, TF-IDF can be a good score to find these meaningful concepts. I agree that TF can be meaningful measure for bigrams or trigrams as well.  
  
  ---
  
  