Answers.Body: <p>Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it.</p>
  
  <p>Word embeddings learn a multi-dimensional context. What exactly the context is of each dimension of the embedding is something of a mystery and simply emerges from the learning. The larger the number of dimensions, the greater the possibility that some combination of the dimensions will represent the sweetness context, but it might be quite hard to tease out.</p>
  
  <p>So you can attach the idea of sweetness to one member of a one-hot structure, but it must necessarily be a part of a rules-based approach. Embeddings, when they are working well, will not need the rules.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-25T13:54:21.587Z
Answers.LastActivityDate: 2018-02-25T13:54:21.587Z
Answers.UserDisplayName: Colin Beckingham
Answers.Body: <p>Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences:</p>
  
  <blockquote>
    <p>The desk has a book on it.</p>
  </blockquote>
  
  <p>and</p>
  
  <blockquote>
    <p>The table has a book on it.</p>
  </blockquote>
  
  <p>These two sentences are almost identical in meaning. If we were to using word embeddings, the vectors 'desk' and 'table' would be very close together. The fact that these two sentences are similar becomes implicit with embeddings.</p>
  
  <p>But if we were to use one-hot vectors, the distance between the two vectors would be the same distance between 'desk' and 'cat' or 'table' and 'book'. So now the network must learn that these sentences may entail the same thing on top of the original task. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-25T14:42:04.103Z
Answers.LastActivityDate: 2018-02-25T14:42:04.103Z
Answers.UserDisplayName: Daniel
Body: <p>How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer?</p>
  <p>I understand how the word embedding carries some semantic meaning, but it seems that this information would get &quot;squashed&quot; by the activation function, leaving only a scalar value and as many different vectors could yield the same result, I would guess that the information is more or less lost.</p>
  <p>Could anyone bring me insights as to why a neural network may utilize the information contained in a word embedding?</p>
  
Tags: <neural-networks><machine-learning><natural-language-processing><word-embedding><embeddings>
Title: What is the intuition behind how word embeddings bring information to a neural network?
UserDisplayName: silkAdmin
ThreadText: # What is the intuition behind how word embeddings bring information to a neural network?
  
  (Asked by: username_0 on 2018-02-23)
  
  How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer?
  
  I understand how the word embedding carries some semantic meaning, but it seems that this information would get "squashed" by the activation function, leaving only a scalar value and as many different vectors could yield the same result, I would guess that the information is more or less lost.
  
  Could anyone bring me insights as to why a neural network may utilize the information contained in a word embedding?
  
  **Tags:** neural-networks, machine-learning, natural-language-processing, word-embedding, embeddings
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-02-25. Score: 5
  
  Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it.
  
  Word embeddings learn a multi-dimensional context. What exactly the context is of each dimension of the embedding is something of a mystery and simply emerges from the learning. The larger the number of dimensions, the greater the possibility that some combination of the dimensions will represent the sweetness context, but it might be quite hard to tease out.
  
  So you can attach the idea of sweetness to one member of a one-hot structure, but it must necessarily be a part of a rules-based approach. Embeddings, when they are working well, will not need the rules.
  
  
  ---
  
  ### Answer by username_2 on 2018-02-25. Score: 3
  
  Adding to username_1's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences:
  
  > The desk has a book on it.
  
  and
  
  > The table has a book on it.
  
  These two sentences are almost identical in meaning. If we were to using word embeddings, the vectors 'desk' and 'table' would be very close together. The fact that these two sentences are similar becomes implicit with embeddings.
  
  But if we were to use one-hot vectors, the distance between the two vectors would be the same distance between 'desk' and 'cat' or 'table' and 'book'. So now the network must learn that these sentences may entail the same thing on top of the original task.
  
  
  ---
  
  