Answers.Body: <p>An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is &quot;<em>context_size * embedding_dim</em>&quot; or &quot;<em>n * embedding_dims</em>&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-12T12:55:00.293Z
Answers.LastActivityDate: 2021-11-13T09:11:18.710Z
Answers.LastEditDate: 2021-11-13T09:11:18.710Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py" rel="nofollow noreferrer">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py</a></p>
  <p>Where we try to create an n-gram language model. However, there's something I don't understand.</p>
  <pre><code>class NGramLanguageModeler(nn.Module):
  
      def __init__(self, vocab_size, embedding_dim, context_size):
          super(NGramLanguageModeler, self).__init__()
          self.embeddings = nn.Embedding(vocab_size, embedding_dim)
          self.linear1 = nn.Linear(context_size * embedding_dim, 128)
          self.linear2 = nn.Linear(128, vocab_size)
  </code></pre>
  <p>at <code>self.linear1 = nn.Linear(context_size * embedding_dim, 128)</code> why did we multiply embedding_dim with context_size? Isn't the embedding_dim input size? So why do we multiply it by the context size?</p>
  
Tags: <natural-language-processing><pytorch><word-embedding>
Title: Why do we multipy context_size with embedding_dim? (PyTorch)
UserDisplayName: J.Smith
ThreadText: # Why do we multipy context_size with embedding_dim? (PyTorch)
  
  (Asked by: username_0 on 2021-11-12)
  
  I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py
  Where we try to create an n-gram language model. However, there's something I don't understand.
  class NGramLanguageModeler(nn.Module):
  
      def __init__(self, vocab_size, embedding_dim, context_size):
          super(NGramLanguageModeler, self).__init__()
          self.embeddings = nn.Embedding(vocab_size, embedding_dim)
          self.linear1 = nn.Linear(context_size * embedding_dim, 128)
          self.linear2 = nn.Linear(128, vocab_size)
  
  at self.linear1 = nn.Linear(context_size * embedding_dim, 128) why did we multiply embedding_dim with context_size? Isn't the embedding_dim input size? So why do we multiply it by the context size?
  
  **Tags:** natural-language-processing, pytorch, word-embedding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-12. Score: 3 | Accepted answer
  
  An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "*context\_size \* embedding\_dim*" or "*n \* embedding\_dims*"
  
  
  ---
  
  