<p>Input vector contains two concatenated parts. The low part represents the current word:</p>

<blockquote>
  <p>word in time t encoded using 1-of-N coding [...] - size of vector x is
  equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]</p>
</blockquote>

<p>where, as you said, 1-of-N means (see <a href="https://www.quora.com/What-is-1-of-V-coding-with-relation-to-the-neural-network-language-model" rel="nofollow noreferrer">here</a>, 1-of-V):</p>

<blockquote>
  <p>If you have a fixed-size vocabulary of symbols with V members in
  total, each input symbol can be coded as a vector of size V with all
  zeros except for the element corresponding to the symbol's order in
  the vocabulary, which gets a 1.</p>
</blockquote>

<p>The high part of the input vector represents the current context:</p>

<blockquote>
  <p>and previous context layer [...] Size of context (hidden) layer s is
  usually 30 - 500 hidden units.</p>
  
  <p>For initialization, s(0) can be set to vector of small values, like
  0.1</p>
</blockquote>

<p>the article includes this expression:</p>

<blockquote>
  <p>x(t) = w(t) + s(t - 1)</p>
</blockquote>

<p>that I think better to write as:</p>

<blockquote>
  <p>x(t) = w(t) || s(t - 1)</p>
</blockquote>

<p>to made more visible the concatenation.</p>

<p>Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:</p>

<blockquote>
  <p>we merge all words that occur less often than a threshold (in the
  training text) into a special rare token.</p>
</blockquote>

CC BY-SA 3.0
2018-02-15T16:26:25.643Z
so `x(t) = w(t) + s(t - 1)` is in fact concatenation of 2 vectors, rather then addition?
Ziemo
CC BY-SA 3.0
2018-02-15T16:27:40.033Z
yes, it is. "w" and "s" have very different sizes, no addition (sum componentwise) possible.
pasaba por aqui
CC BY-SA 3.0
2018-02-13T09:32:36.050Z
2018-02-13T10:45:15.643Z
2018-02-13T10:45:15.643Z
pasaba por aqui
<p>I'm reading <em>"<a href="http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf" rel="nofollow noreferrer">Recurrent neural network based language model</a>"</em> of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding <span class="math-container">$w(t)$</span> is obtained:</p>

<p><a href="https://i.sstatic.net/6i54I.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6i54I.png" alt="enter image description here"></a></p>

<p>The reason I wonder is that in the classic "<em><a href="http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf" rel="nofollow noreferrer">A Neural Probabilistic Language Model</a></em>" Bengio et al. (2003) - they used separate embedding vector for representing each word and it was somehow "semi-layer", meaning - it haven't contains non-linearity, but they did update word embeddings during the back-propagation.</p>

<p>In Mikolov approach though, I assume they used simple one-hot vector, where each feature represent presence of each word. If we represent that's way single word input (like was in the Mikolov's paper) - that vector become all-zeros except single one.</p>

<p>Is that correct?</p>

<natural-language-processing><recurrent-neural-networks><word-embedding><papers>
How is the word embedding represented in the paper "Recurrent neural network based language model"?
Ziemo
# How is the word embedding represented in the paper "Recurrent neural network based language model"?

(Asked by: username_0 on 2018-02-12)

I'm reading *"[Recurrent neural network based language model](http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf)"* of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding $w(t)$ is obtained:

[![enter image description here](https://i.sstatic.net/6i54I.png)](https://i.sstatic.net/6i54I.png)

The reason I wonder is that in the classic "*[A Neural Probabilistic Language Model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)*" Bengio et al. (2003) - they used separate embedding vector for representing each word and it was somehow "semi-layer", meaning - it haven't contains non-linearity, but they did update word embeddings during the back-propagation.

In Mikolov approach though, I assume they used simple one-hot vector, where each feature represent presence of each word. If we represent that's way single word input (like was in the Mikolov's paper) - that vector become all-zeros except single one.

Is that correct?

**Tags:** natural-language-processing, recurrent-neural-networks, word-embedding, papers


---

## 1 answers

### Answer by username_1 on 2018-02-13. Score: 2 | Accepted answer

Input vector contains two concatenated parts. The low part represents the current word:

> word in time t encoded using 1-of-N coding [...] - size of vector x is
> equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]

where, as you said, 1-of-N means (see [here](https://www.quora.com/What-is-1-of-V-coding-with-relation-to-the-neural-network-language-model), 1-of-V):

> If you have a fixed-size vocabulary of symbols with V members in
> total, each input symbol can be coded as a vector of size V with all
> zeros except for the element corresponding to the symbol's order in
> the vocabulary, which gets a 1.

The high part of the input vector represents the current context:

> and previous context layer [...] Size of context (hidden) layer s is
> usually 30 - 500 hidden units.
>
> For initialization, s(0) can be set to vector of small values, like
> 0.1

the article includes this expression:

> x(t) = w(t) + s(t - 1)

that I think better to write as:

> x(t) = w(t) || s(t - 1)

to made more visible the concatenation.

Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:

> we merge all words that occur less often than a threshold (in the
> training text) into a special rare token.

> **Comments:**
> **username_0**: so `x(t) = w(t) + s(t - 1)` is in fact concatenation of 2 vectors, rather then addition?  
> **username_1**: yes, it is. "w" and "s" have very different sizes, no addition (sum componentwise) possible.  

---

