Answers.Body: <p>If you look at the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/sparse.html#Embedding" rel="nofollow noreferrer">source code</a> of PyTorch's <code>Embedding</code> layer, you can see that it defines a variable called <code>self.weight</code> as a <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" rel="nofollow noreferrer"><code>Parameter</code></a>, which is a subclass of the <code>Tensor</code>, i.e. something that can be changed by gradient descent (you can do that by setting the parameter <a href="https://pytorch.org/docs/stable/notes/autograd.html#locally-disable-grad-doc" rel="nofollow noreferrer"><code>requires_grad</code></a> of the <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" rel="nofollow noreferrer"><code>Parameter</code></a> to <code>True</code>). In other words, the <code>Embedding</code> layer is not just a look-up table, but it's a layer where you have parameters (i.e. the embeddings, which are stored in <code>self.weight</code>) that can also be learnable. You can also initialize these embeddings (i.e. the <code>self.weight</code> parameter) from pre-trained ones using <code>Embedding</code>'s method <code>from_pretrained</code>. In this case, you should set <code>require_grad</code> to <code>False</code>.</p>
  <p>Generally, one can define an embedding layer <span class="math-container">$\mathcal{f}$</span> as a function that receives the raw inputs <span class="math-container">$\mathbf{i}$</span> (e.g. in the case of word embeddings, the raw inputs might be integers: one for each word) and transforms them to embeddings <span class="math-container">$\mathbf{e}$</span>, which can be statically defined (e.g. from pre-trained embeddings or hardcoded), randomly initialized and/or learnable (during the training of the neural network). In other words, <span class="math-container">$f(\mathbf{i}) = \mathbf{e}$</span>. So, that's why we pass <span class="math-container">$f(\mathbf{i})$</span>, i.e. the embeddings, rather than <span class="math-container">$\mathbf{i}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-13T22:41:22.243Z
Answers.Text: I will maybe improve this answer later by referring to word2vec.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-13T15:36:40.153Z
Answers.LastActivityDate: 2021-12-13T22:36:59.943Z
Answers.LastEditDate: 2021-12-13T22:36:59.943Z
Answers.UserDisplayName: nbro
Body: <p>I am reading about RNN encoders. I came across the following line from <a href="https://github.com/IntelLabs/distiller/blob/master/examples/word_language_model/model.py" rel="nofollow noreferrer">this</a> code. And I am facing difficulty in understanding the theoretical details regarding it.</p>
  <pre><code>emb = self.drop(self.encoder(input))
  </code></pre>
  <p>The <code>input</code> is a tensor of shape <span class="math-container">$[32, 100]$</span>. Here 32 is the batch size and 100 is the length of the sentence. Hundred elements are indices to the words (from the dictionary) that are used in the sentence. We can observe that the output <code>emb</code> is later passed to the rnn (LSTM/GRU) layer.</p>
  <pre><code>output, hidden = self.rnn(emb, hidden)
  </code></pre>
  <p>So, to me, it looks like that <code>self.encoder</code> is the <em>necessary step</em> while using the RNN encoder. So, I am interested in what it actually does.</p>
  <p>When we see about <code>self.encoder</code>, it is an <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html" rel="nofollow noreferrer">Embedding layer</a>. The description for this layer is as follows</p>
  <blockquote>
  <p>A <strong>simple lookup table</strong> that stores embeddings of a fixed dictionary and
  size.</p>
  <p>This <strong>module</strong> is often used to store word embeddings and retrieve them
  using indices. The input to the <strong>module</strong> is a list of indices, and the
  output is the corresponding <strong>word embeddings</strong>.</p>
  </blockquote>
  <p>When we see about <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html" rel="nofollow noreferrer"><code>self.drop</code></a>, it randomly keeps zero in the embeddings.</p>
  <blockquote>
  <p>During training, randomly zeroes some of the elements of the input
  tensor with probability p using samples from a Bernoulli distribution.
  Each channel will be zeroed out independently on every forward call.</p>
  </blockquote>
  <p>The outputs for both <code>self.encoder(input)</code> and <code>self.drop(self.encoder(input))</code> are <span class="math-container">$[32, 100, 3000]$</span>.</p>
  <p>I have doubt(s) on the bolded parts of the description of the Embedding layer. The description is saying the Embedding layer uses/contains(?) a lookup table. The description says Embedding layer <em>stores and retrieves</em> word embeddings.</p>
  <p>The doubts are</p>
  <ol>
  <li><p>Generally, does an embedding layer <em>calculate</em> word embeddings or just store and retrieve them from the table? If it does not calculate them, then who will calculate the embeddings? If you can also comment on the specifics of PyTorch, I would appreciate it.</p>
  </li>
  <li><p>What exactly is an embedding layer? Is it a collection of neurons or any other?</p>
  </li>
  </ol>
  
Tags: <deep-learning><recurrent-neural-networks><pytorch><word-embedding>
Title: What exactly is embedding layer used in RNN encoders?
UserDisplayName: hanugm
ThreadText: # What exactly is embedding layer used in RNN encoders?
  
  (Asked by: username_0 on 2021-12-11)
  
  I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it.
  emb = self.drop(self.encoder(input))
  
  The input is a tensor of shape $[32, 100]$. Here 32 is the batch size and 100 is the length of the sentence. Hundred elements are indices to the words (from the dictionary) that are used in the sentence. We can observe that the output emb is later passed to the rnn (LSTM/GRU) layer.
  output, hidden = self.rnn(emb, hidden)
  
  So, to me, it looks like that self.encoder is the necessary step while using the RNN encoder. So, I am interested in what it actually does.
  When we see about self.encoder, it is an Embedding layer. The description for this layer is as follows
  
  A simple lookup table that stores embeddings of a fixed dictionary and
  size.
  This module is often used to store word embeddings and retrieve them
  using indices. The input to the module is a list of indices, and the
  output is the corresponding word embeddings.
  
  When we see about self.drop, it randomly keeps zero in the embeddings.
  
  During training, randomly zeroes some of the elements of the input
  tensor with probability p using samples from a Bernoulli distribution.
  Each channel will be zeroed out independently on every forward call.
  
  The outputs for both self.encoder(input) and self.drop(self.encoder(input)) are $[32, 100, 3000]$.
  I have doubt(s) on the bolded parts of the description of the Embedding layer. The description is saying the Embedding layer uses/contains(?) a lookup table. The description says Embedding layer stores and retrieves word embeddings.
  The doubts are
  
  Generally, does an embedding layer calculate word embeddings or just store and retrieve them from the table? If it does not calculate them, then who will calculate the embeddings? If you can also comment on the specifics of PyTorch, I would appreciate it.
  
  What exactly is an embedding layer? Is it a collection of neurons or any other?
  
  **Tags:** deep-learning, recurrent-neural-networks, pytorch, word-embedding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-13. Score: 1
  
  If you look at the [source code](https://pytorch.org/docs/stable/_modules/torch/nn/modules/sparse.html#Embedding) of PyTorch's `Embedding` layer, you can see that it defines a variable called `self.weight` as a [`Parameter`](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html), which is a subclass of the `Tensor`, i.e. something that can be changed by gradient descent (you can do that by setting the parameter [`requires_grad`](https://pytorch.org/docs/stable/notes/autograd.html#locally-disable-grad-doc) of the [`Parameter`](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html) to `True`). In other words, the `Embedding` layer is not just a look-up table, but it's a layer where you have parameters (i.e. the embeddings, which are stored in `self.weight`) that can also be learnable. You can also initialize these embeddings (i.e. the `self.weight` parameter) from pre-trained ones using `Embedding`'s method `from_pretrained`. In this case, you should set `require_grad` to `False`.
  
  Generally, one can define an embedding layer $\mathcal{f}$ as a function that receives the raw inputs $\mathbf{i}$ (e.g. in the case of word embeddings, the raw inputs might be integers: one for each word) and transforms them to embeddings $\mathbf{e}$, which can be statically defined (e.g. from pre-trained embeddings or hardcoded), randomly initialized and/or learnable (during the training of the neural network). In other words, $f(\mathbf{i}) = \mathbf{e}$. So, that's why we pass $f(\mathbf{i})$, i.e. the embeddings, rather than $\mathbf{i}$.
  
  > **Comments:**
  > **username_1**: I will maybe improve this answer later by referring to word2vec.  
  
  ---
  
  