Answers.Body: <p>This issue was resolved by normalizing the vectors.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-18T09:48:21.243Z
Answers.LastActivityDate: 2017-10-18T09:48:21.243Z
Answers.UserDisplayName: Hartger
Body: <p>I'm working on a content based recommendation engine for ebooks. I create document vectors with 300 features for every ebook using a word2vec model trained on google news and determine recommendations based on the closest vectors. So far I have run tests on a dataset consisting of 200 books from the gutenberg project in four different categories.</p>
  
  <p>I find that a small group of books appears to be recommended a lot more then others. The most recommended book is recommended for over half the dataset. This book is Theaetus, in which the <a href="https://gist.github.com/HartgerV/50ccdb999393eb1fd2b50725e14cd0fa" rel="nofollow noreferrer">most commmon tokens</a> are fairly specific to the book (tokens like socrates, theaetus). I can find no intuitive reason why this book would match so well with over half my dataset. </p>
  
  <p>Is this common behavior when using document vectors to determine similarity? Are there any methods that would reduce this effect?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-09T13:24:34.227Z
Comments.Text: Would that be Theatetus has more occurrence than all other books?
Comments.UserDisplayName: SmallChess
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-09T13:29:35.540Z
Comments.Text: The model purely checks distances between vectors, so occurrence isn't really a factor here.
Comments.UserDisplayName: Hartger
Tags: <natural-language-processing><word-embedding><recommender-system>
Title: Over-exposure of certain items in content based recommendation engine
UserDisplayName: Hartger
ThreadText: # Over-exposure of certain items in content based recommendation engine
  
  (Asked by: username_0 on 2017-10-09)
  
  I'm working on a content based recommendation engine for ebooks. I create document vectors with 300 features for every ebook using a word2vec model trained on google news and determine recommendations based on the closest vectors. So far I have run tests on a dataset consisting of 200 books from the gutenberg project in four different categories.
  
  I find that a small group of books appears to be recommended a lot more then others. The most recommended book is recommended for over half the dataset. This book is Theaetus, in which the [most commmon tokens](https://gist.github.com/HartgerV/50ccdb999393eb1fd2b50725e14cd0fa) are fairly specific to the book (tokens like socrates, theaetus). I can find no intuitive reason why this book would match so well with over half my dataset.
  
  Is this common behavior when using document vectors to determine similarity? Are there any methods that would reduce this effect?
  
  **Tags:** natural-language-processing, word-embedding, recommender-system
  > **Comments:**
  > **username_1**: Would that be Theatetus has more occurrence than all other books?  
  > **username_0**: The model purely checks distances between vectors, so occurrence isn't really a factor here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2017-10-18. Score: 0 | Accepted answer
  
  This issue was resolved by normalizing the vectors.
  
  
  ---
  
  