Answers.Body: <p>Input vector contains two concatenated parts. The low part represents the current word:</p>
  
  <blockquote>
    <p>word in time t encoded using 1-of-N coding [...] - size of vector x is
    equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]</p>
  </blockquote>
  
  <p>where, as you said, 1-of-N means (see <a href="https://www.quora.com/What-is-1-of-V-coding-with-relation-to-the-neural-network-language-model" rel="nofollow noreferrer">here</a>, 1-of-V):</p>
  
  <blockquote>
    <p>If you have a fixed-size vocabulary of symbols with V members in
    total, each input symbol can be coded as a vector of size V with all
    zeros except for the element corresponding to the symbol's order in
    the vocabulary, which gets a 1.</p>
  </blockquote>
  
  <p>The high part of the input vector represents the current context:</p>
  
  <blockquote>
    <p>and previous context layer [...] Size of context (hidden) layer s is
    usually 30 - 500 hidden units.</p>
    
    <p>For initialization, s(0) can be set to vector of small values, like
    0.1</p>
  </blockquote>
  
  <p>the article includes this expression:</p>
  
  <blockquote>
    <p>x(t) = w(t) + s(t - 1)</p>
  </blockquote>
  
  <p>that I think better to write as:</p>
  
  <blockquote>
    <p>x(t) = w(t) || s(t - 1)</p>
  </blockquote>
  
  <p>to made more visible the concatenation.</p>
  
  <p>Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:</p>
  
  <blockquote>
    <p>we merge all words that occur less often than a threshold (in the
    training text) into a special rare token.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-15T16:26:25.643Z
Answers.Text: so `x(t) = w(t) + s(t - 1)` is in fact concatenation of 2 vectors, rather then addition?
Answers.UserDisplayName: Ziemo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-15T16:27:40.033Z
Answers.Text: yes, it is. "w" and "s" have very different sizes, no addition (sum componentwise) possible.
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-13T09:32:36.050Z
Answers.LastActivityDate: 2018-02-13T10:45:15.643Z
Answers.LastEditDate: 2018-02-13T10:45:15.643Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>I'm reading <em>"<a href="http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf" rel="nofollow noreferrer">Recurrent neural network based language model</a>"</em> of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding <span class="math-container">$w(t)$</span> is obtained:</p>
  
  <p><a href="https://i.sstatic.net/6i54I.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6i54I.png" alt="enter image description here"></a></p>
  
  <p>The reason I wonder is that in the classic "<em><a href="http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf" rel="nofollow noreferrer">A Neural Probabilistic Language Model</a></em>" Bengio et al. (2003) - they used separate embedding vector for representing each word and it was somehow "semi-layer", meaning - it haven't contains non-linearity, but they did update word embeddings during the back-propagation.</p>
  
  <p>In Mikolov approach though, I assume they used simple one-hot vector, where each feature represent presence of each word. If we represent that's way single word input (like was in the Mikolov's paper) - that vector become all-zeros except single one.</p>
  
  <p>Is that correct?</p>
  
Tags: <natural-language-processing><recurrent-neural-networks><word-embedding><papers>
Title: How is the word embedding represented in the paper "Recurrent neural network based language model"?
UserDisplayName: Ziemo
ThreadText: # How is the word embedding represented in the paper "Recurrent neural network based language model"?
  
  (Asked by: username_0 on 2018-02-12)
  
  I'm reading *"[Recurrent neural network based language model](http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf)"* of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding $w(t)$ is obtained:
  
  [![enter image description here](https://i.sstatic.net/6i54I.png)](https://i.sstatic.net/6i54I.png)
  
  The reason I wonder is that in the classic "*[A Neural Probabilistic Language Model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)*" Bengio et al. (2003) - they used separate embedding vector for representing each word and it was somehow "semi-layer", meaning - it haven't contains non-linearity, but they did update word embeddings during the back-propagation.
  
  In Mikolov approach though, I assume they used simple one-hot vector, where each feature represent presence of each word. If we represent that's way single word input (like was in the Mikolov's paper) - that vector become all-zeros except single one.
  
  Is that correct?
  
  **Tags:** natural-language-processing, recurrent-neural-networks, word-embedding, papers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-02-13. Score: 2 | Accepted answer
  
  Input vector contains two concatenated parts. The low part represents the current word:
  
  > word in time t encoded using 1-of-N coding [...] - size of vector x is
  > equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]
  
  where, as you said, 1-of-N means (see [here](https://www.quora.com/What-is-1-of-V-coding-with-relation-to-the-neural-network-language-model), 1-of-V):
  
  > If you have a fixed-size vocabulary of symbols with V members in
  > total, each input symbol can be coded as a vector of size V with all
  > zeros except for the element corresponding to the symbol's order in
  > the vocabulary, which gets a 1.
  
  The high part of the input vector represents the current context:
  
  > and previous context layer [...] Size of context (hidden) layer s is
  > usually 30 - 500 hidden units.
  >
  > For initialization, s(0) can be set to vector of small values, like
  > 0.1
  
  the article includes this expression:
  
  > x(t) = w(t) + s(t - 1)
  
  that I think better to write as:
  
  > x(t) = w(t) || s(t - 1)
  
  to made more visible the concatenation.
  
  Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:
  
  > we merge all words that occur less often than a threshold (in the
  > training text) into a special rare token.
  
  > **Comments:**
  > **username_0**: so `x(t) = w(t) + s(t - 1)` is in fact concatenation of 2 vectors, rather then addition?  
  > **username_1**: yes, it is. "w" and "s" have very different sizes, no addition (sum componentwise) possible.  
  
  ---
  
  