Answers.Body: <p><strong>Doc2Vec</strong></p>
  
  <p>Doc2Vec comes to mind, here's the <a href="https://cs.stanford.edu/~quocle/paragraph_vector.pdf" rel="nofollow noreferrer">original publication</a>. The approach has been shown to be very successful for certain NLP-based problems, though I haven't personally used it for a project yet. </p>
  
  <p>There are a number of implementations of Doc2Vec. If you're using Python, one to look at is <a href="https://radimrehurek.com/gensim/models/doc2vec.html" rel="nofollow noreferrer">gensim</a>. </p>
  
  <p><strong>Word2Vec</strong></p>
  
  <p>Word2Vec is similar to Doc2Vec and perhaps more in line with what you're looking for. Here's the <a href="https://arxiv.org/pdf/1301.3781.pdf" rel="nofollow noreferrer">original publication</a>, and <a href="https://www.cs.bgu.ac.il/~yoavg/publications/negative-sampling.pdf" rel="nofollow noreferrer">another publication</a> that does a nice job explaining it further. </p>
  
  <p>Tensorflow has a <a href="https://www.tensorflow.org/tutorials/word2vec" rel="nofollow noreferrer">tutorial</a> for setting up a Word2Vec model. Gensim also has a <a href="https://radimrehurek.com/gensim/models/word2vec.html" rel="nofollow noreferrer">Word2Vec implementation</a>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-23T10:25:28.713Z
Answers.Text: Word2Vec (and probably Doc2Vec) can be used to learn embeddings, but are they used directly to compare "concept similarity"? Or do you need to learn the embeddings first and then use some kind of "similarity metric" between embeddings? Also, why would you learn embeddings, and what do they represent? This is not clear from this answer. So, I think you should edit this answer to clarify that. I am familiar with these concepts, so I have an idea of what the answers to my questions here are, but I suspect that, in this form, this answer doesn't directly answer the original question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T17:28:24.627Z
Answers.Text: @nbro I wrote this answer five years ago. While I stand by it for the the reasons that 1. Word2Vec embeddings enable one to do concept arithmetic over learned embeddings, and 2. all the examples described in the OPs question would be embedded closely to one another in a Word2Vec embedding (and this distance could be measured and normalized to the range [0,1] as needed), I suspect that there are better solutions now. If you feel these two points aren't sufficient explanation for the community, feel free to add your own answer with an update.
Answers.UserDisplayName: Greenstick
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-26T20:29:52.603Z
Answers.LastActivityDate: 2017-09-27T04:48:39.333Z
Answers.LastEditDate: 2017-09-27T04:48:39.333Z
Answers.UserDisplayName: Greenstick
Answers.Body: <p>In machine translation, there is a widely used BLEU score ( <a href="https://en.wikipedia.org/wiki/BLEU" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/BLEU</a> ).
  It simply counts the matching n-grams between two segments of text
  and returns a 0-1 score based on that.</p>
  <p>The problem with this method is that it would give the same score to
  pairs &quot;It is hot&quot;/&quot;It is cold&quot; and &quot;It is hot&quot; / &quot;It is warm&quot;.
  There is no nuance for spelling or synonyms: for each word, you either have a literal match or you don't.</p>
  <p>A recent refinement to BLEU is BLEURT (<a href="https://arxiv.org/abs/2004.04696" rel="nofollow noreferrer">https://arxiv.org/abs/2004.04696</a>).
  The key idea is to also look at BERT embedding for both segments of text,
  which now allows you to notice that &quot;hot/warm&quot; are more similar than &quot;hot/cold&quot;, and produce a more nuanced score.</p>
  <p>This is all specific to machine translation evaluation , and may or may not work for your specific application.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-23T21:30:01.770Z
Answers.LastActivityDate: 2022-01-23T21:30:01.770Z
Answers.UserDisplayName: user2907934
Body: <p>Are there any algorithms (or software libraries) that can be used to detect the similarity of concepts in text, regardless of articulation, grammar, synonyms, etc.?</p>
  <p>For example, these phrases:</p>
  <blockquote>
  <p>Outside, it is warm.</p>
  <p>Outside, it is hot.</p>
  <p>Outside, it is not cold.</p>
  <p>It is not cold outside.</p>
  </blockquote>
  <p>Should be similar to this phrase:</p>
  <blockquote>
  <p>It is warm outside.</p>
  </blockquote>
  <p>Ideally, the algorithm (or software) would be capable of generating a score from 0 to 1, based on the concept similarity. The goal is to use this algorithm or software to map a large number of statements to a single, similar original statement. It is for this mapping of a given statement to the original statement that the aforementioned similarity score would be generated.</p>
  <p>Does such an algorithm (or software) already exist?</p>
  
Tags: <neural-networks><natural-language-processing><algorithm-request>
Title: Are there any algorithms that can measure the concept similarity between texts?
UserDisplayName: omega
ThreadText: # Are there any algorithms that can measure the concept similarity between texts?
  
  (Asked by: username_0 on 2017-09-26)
  
  Are there any algorithms (or software libraries) that can be used to detect the similarity of concepts in text, regardless of articulation, grammar, synonyms, etc.?
  
  For example, these phrases:
  
  > Outside, it is warm.
  >
  > Outside, it is hot.
  >
  > Outside, it is not cold.
  >
  > It is not cold outside.
  
  Should be similar to this phrase:
  
  > It is warm outside.
  
  Ideally, the algorithm (or software) would be capable of generating a score from 0 to 1, based on the concept similarity. The goal is to use this algorithm or software to map a large number of statements to a single, similar original statement. It is for this mapping of a given statement to the original statement that the aforementioned similarity score would be generated.
  
  Does such an algorithm (or software) already exist?
  
  **Tags:** neural-networks, natural-language-processing, algorithm-request
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2017-09-26. Score: 5
  
  **Doc2Vec**
  
  Doc2Vec comes to mind, here's the [original publication](https://cs.stanford.edu/~quocle/paragraph_vector.pdf). The approach has been shown to be very successful for certain NLP-based problems, though I haven't personally used it for a project yet.
  
  There are a number of implementations of Doc2Vec. If you're using Python, one to look at is [gensim](https://radimrehurek.com/gensim/models/doc2vec.html).
  
  **Word2Vec**
  
  Word2Vec is similar to Doc2Vec and perhaps more in line with what you're looking for. Here's the [original publication](https://arxiv.org/pdf/1301.3781.pdf), and [another publication](https://www.cs.bgu.ac.il/~yoavg/publications/negative-sampling.pdf) that does a nice job explaining it further.
  
  Tensorflow has a [tutorial](https://www.tensorflow.org/tutorials/word2vec) for setting up a Word2Vec model. Gensim also has a [Word2Vec implementation](https://radimrehurek.com/gensim/models/word2vec.html).
  
  > **Comments:**
  > **username_2**: Word2Vec (and probably Doc2Vec) can be used to learn embeddings, but are they used directly to compare "concept similarity"? Or do you need to learn the embeddings first and then use some kind of "similarity metric" between embeddings? Also, why would you learn embeddings, and what do they represent? This is not clear from this answer. So, I think you should edit this answer to clarify that. I am familiar with these concepts, so I have an idea of what the answers to my questions here are, but I suspect that, in this form, this answer doesn't directly answer the original question.  
  > **username_1**: @nbro I wrote this answer five years ago. While I stand by it for the the reasons that 1. Word2Vec embeddings enable one to do concept arithmetic over learned embeddings, and 2. all the examples described in the OPs question would be embedded closely to one another in a Word2Vec embedding (and this distance could be measured and normalized to the range [0,1] as needed), I suspect that there are better solutions now. If you feel these two points aren't sufficient explanation for the community, feel free to add your own answer with an update.  
  
  ---
  
  ### Answer by username_3 on 2022-01-23. Score: 0
  
  In machine translation, there is a widely used BLEU score ( <https://en.wikipedia.org/wiki/BLEU> ).
  It simply counts the matching n-grams between two segments of text
  and returns a 0-1 score based on that.
  
  The problem with this method is that it would give the same score to
  pairs "It is hot"/"It is cold" and "It is hot" / "It is warm".
  There is no nuance for spelling or synonyms: for each word, you either have a literal match or you don't.
  
  A recent refinement to BLEU is BLEURT (<https://arxiv.org/abs/2004.04696>).
  The key idea is to also look at BERT embedding for both segments of text,
  which now allows you to notice that "hot/warm" are more similar than "hot/cold", and produce a more nuanced score.
  
  This is all specific to machine translation evaluation , and may or may not work for your specific application.
  
  
  ---
  
  