<p>An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is &quot;<em>context_size * embedding_dim</em>&quot; or &quot;<em>n * embedding_dims</em>&quot;</p>

CC BY-SA 4.0
2021-11-12T12:55:00.293Z
2021-11-13T09:11:18.710Z
2021-11-13T09:11:18.710Z
Edoardo Guerriero
<p>I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py" rel="nofollow noreferrer">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py</a></p>
<p>Where we try to create an n-gram language model. However, there's something I don't understand.</p>
<pre><code>class NGramLanguageModeler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(NGramLanguageModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size * embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)
</code></pre>
<p>at <code>self.linear1 = nn.Linear(context_size * embedding_dim, 128)</code> why did we multiply embedding_dim with context_size? Isn't the embedding_dim input size? So why do we multiply it by the context size?</p>

<natural-language-processing><pytorch><word-embedding>
Why do we multipy context_size with embedding_dim? (PyTorch)
J.Smith
# Why do we multipy context_size with embedding_dim? (PyTorch)

(Asked by: username_0 on 2021-11-12)

I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py
Where we try to create an n-gram language model. However, there's something I don't understand.
class NGramLanguageModeler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(NGramLanguageModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size * embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)

at self.linear1 = nn.Linear(context_size * embedding_dim, 128) why did we multiply embedding_dim with context_size? Isn't the embedding_dim input size? So why do we multiply it by the context size?

**Tags:** natural-language-processing, pytorch, word-embedding


---

## 1 answers

### Answer by username_1 on 2021-11-12. Score: 3 | Accepted answer

An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "*context\_size \* embedding\_dim*" or "*n \* embedding\_dims*"


---

