Answers.Body: <p>You can use a model to create rich embeddings for example: <a href="https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens" rel="nofollow noreferrer">sentence transformers</a> and then use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html" rel="nofollow noreferrer">cosine similarity distance from sklearn</a> with a threshold (at least 0.6) to create clusters of semantically-close document</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T15:59:54.900Z
Answers.Text: Interesting. I should try it :)
Answers.UserDisplayName: Eitan Rosati
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T12:59:42.770Z
Answers.LastActivityDate: 2022-06-15T12:59:42.770Z
Answers.UserDisplayName: Saeron X
Body: <p>I have a database of books. Each book have a list of categories that describe the genre/topics of the book (I use Python models).</p>
  <p>Most of the time, the categories in the list are composed from 1-3 words.</p>
  <p>Examples of a book category list:</p>
  <pre><code>['Children', 'Flour mills', 'Jealousy', 'Nannies', 'Child labor', 'Conduct of life'],
  [&quot;Children's stories&quot;, 'Christian life'],
  ['Children', 'Brothers and sisters', 'Conduct of life', 'Cheerfulness', 'Christian life'],
  ['Fugitive slaves', 'African Americans', 'Slavery', 'Plantation life', 'Slaves', 'Christian life', 'Cruelty']
  </code></pre>
  <p>I want to create/use an algorithm to compare the books and find similarity between 2 books using NLP/machine learning models.</p>
  <p>The categories are not well defined and tend to change. For example, there can be a category of <code>'story'</code> and other called <code>'stories'</code> category (since the text in the system don't saved categories and use a open text box).</p>
  <p>So far I tried 2 algorithms:</p>
  <ul>
  <li>cosine similarity with WordNet - split the category to bag of words and check if each word have synonym in the other book lists.</li>
  <li>Check the similarity using the <code>nlp</code> model of the spacy library (Python) - distance algorithm.</li>
  </ul>
  <p>So far I used WordNet model from the <code>nltk</code> package and <code>spacy</code>
  had problem with those two algorithms because when the algorithm compare a categories that contain 2 or 3 words the results wasn't accurate and each of them had specific problems.</p>
  <p>Which algorithm and models (in Python), can I use to compare between the books that can handle strings that contain 2 or 3 words?</p>
  <p>B.w is the first time I ask here. If you need more details about the database or what I did so far please tell me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T22:39:16.817Z
Comments.Text: One thing is not clear to me. Do you want to compute the similarity of books by only considering these list of categories? From your description, this seems to be the case. So, what is your actual question?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T23:17:11.130Z
Comments.Text: Hey. Firstable thanks for your comment. As i described the categories are not solid - since they was added as an open text. I need a tool that compare and find similarity between those strings(we can also treat them as a short phrase). For example can be categories like `child`, `children` and `children story`. All of them are about children but the algorithms i mentioned before can't compare between a string of 2-3 words in a good way. So i search a techniques to find similarity/compare between those strings
Comments.UserDisplayName: Eitan Rosati
Tags: <machine-learning><natural-language-processing><python><algorithm-request><spacy>
Title: Compare Strings composed from 2-3 words using NLP/ML(Python)
UserDisplayName: Eitan Rosati
ThreadText: # Compare Strings composed from 2-3 words using NLP/ML(Python)
  
  (Asked by: username_0 on 2022-02-26)
  
  I have a database of books. Each book have a list of categories that describe the genre/topics of the book (I use Python models).
  
  Most of the time, the categories in the list are composed from 1-3 words.
  
  Examples of a book category list:
  
  ```
  ['Children', 'Flour mills', 'Jealousy', 'Nannies', 'Child labor', 'Conduct of life'],
  ["Children's stories", 'Christian life'],
  ['Children', 'Brothers and sisters', 'Conduct of life', 'Cheerfulness', 'Christian life'],
  ['Fugitive slaves', 'African Americans', 'Slavery', 'Plantation life', 'Slaves', 'Christian life', 'Cruelty']
  
  ```
  
  I want to create/use an algorithm to compare the books and find similarity between 2 books using NLP/machine learning models.
  
  The categories are not well defined and tend to change. For example, there can be a category of `'story'` and other called `'stories'` category (since the text in the system don't saved categories and use a open text box).
  
  So far I tried 2 algorithms:
  
  * cosine similarity with WordNet - split the category to bag of words and check if each word have synonym in the other book lists.
  * Check the similarity using the `nlp` model of the spacy library (Python) - distance algorithm.
  
  So far I used WordNet model from the `nltk` package and `spacy`
  had problem with those two algorithms because when the algorithm compare a categories that contain 2 or 3 words the results wasn't accurate and each of them had specific problems.
  
  Which algorithm and models (in Python), can I use to compare between the books that can handle strings that contain 2 or 3 words?
  
  B.w is the first time I ask here. If you need more details about the database or what I did so far please tell me.
  
  **Tags:** machine-learning, natural-language-processing, python, algorithm-request, spacy
  
  > **Comments:**
  > **username_1**: One thing is not clear to me. Do you want to compute the similarity of books by only considering these list of categories? From your description, this seems to be the case. So, what is your actual question?  
  > **username_0**: Hey. Firstable thanks for your comment. As i described the categories are not solid - since they was added as an open text. I need a tool that compare and find similarity between those strings(we can also treat them as a short phrase). For example can be categories like `child`, `children` and `children story`. All of them are about children but the algorithms i mentioned before can't compare between a string of 2-3 words in a good way. So i search a techniques to find similarity/compare between those strings  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-15. Score: 1
  
  You can use a model to create rich embeddings for example: [sentence transformers](https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) and then use [cosine similarity distance from sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html) with a threshold (at least 0.6) to create clusters of semantically-close document
  
  > **Comments:**
  > **username_0**: Interesting. I should try it :)  
  
  ---
  
  