Answers.Body: <p>Well actually the dictionary does not contain entire words but rather <a href="https://en.m.wikipedia.org/wiki/Byte_pair_encoding" rel="nofollow noreferrer">byte pair encodings</a>. At the end of chapter 3 of the <a href="https://arxiv.org/pdf/1608.05859.pdf" rel="nofollow noreferrer">paper</a> that you listed they briefly mention that around 85-90% of the subwords are shared between english and french (see also Table 1). So in this case sharing the weights between the source embedding layer and the target embedding layer makes a lot of sense.</p>
  <p>I think that if you are in a different setting (e.g. english to chinese), where your final layer contains <span class="math-container">$n^e + n^f$</span> outputs, then this doesn't make much sense because during decoding you actually want to generate words only from the target language, and training this huge softmax layer might be really tricky and hard to do.</p>
  <p>Now regarding sharing the weights between the <strong>decoder</strong> embedding layer and the <strong>decoder</strong> output layer. I think this is a very nice idea, and you can always do this regardless of the source language or the task altoghether. However, you might want to be careful with the initialization of these weights. You can checkout <a href="https://pi-tau.github.io/posts/transformer/#token-embedding-layer" rel="nofollow noreferrer">this</a> if you want to read more.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T15:56:00.347Z
Answers.LastActivityDate: 2023-07-23T15:56:00.347Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>I found the answer by reading the paper referenced by that section, <a href="https://arxiv.org/pdf/1608.05859.pdf" rel="nofollow noreferrer">Using the output embedding to improve language models</a></p>
  <blockquote>
  <p>Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of
  the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of this model is the union of both the source
  and target vocabularies. In this model, both in the
  encoder and decoder, all subwords are embedded
  in the same duo-lingual space.</p>
  </blockquote>
  <p>It seems like they learned a single embedding matrix (<span class="math-container">$n^e + n^f$</span>) x <span class="math-container">$d^{model}$</span> in dimension.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-06T21:03:43.913Z
Answers.LastActivityDate: 2020-12-06T21:03:43.913Z
Answers.UserDisplayName: user3667125
Body: <p>In the paper <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="nofollow noreferrer">Attention Is All You Need</a>, this section confuses me:</p>
  <blockquote>
  <p>In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear transformation [output of the decoding section]</p>
  </blockquote>
  <p>Shouldn't the weights be different, and not the same? Here is my understanding:</p>
  <p>For simplicity, let us use the English-to-French translation task where we have <span class="math-container">$n^e$</span> number of English words in our dictionary and <span class="math-container">$n^f$</span> number of French words.</p>
  <ul>
  <li><p>In the encoding layer, the input tokens are <span class="math-container">$1$</span> x <span class="math-container">$n^e$</span> one-hot vectors, and are embedded with a <span class="math-container">$n^e$</span> x <span class="math-container">$d^{model}$</span> learned embedding matrix.</p>
  </li>
  <li><p>In the output of the decoding layer, the final step is a linear transformation with weight matrix <span class="math-container">$d^{model}$</span> x <span class="math-container">$n^f$</span>, and then applying softmax to get the probability of each french word, and choosing the french word with the highest probability.</p>
  </li>
  </ul>
  <p>How is it that the <span class="math-container">$n^e$</span> x <span class="math-container">$n^{model}$</span> input embedding matrix share the same weights as the <span class="math-container">$d^{model}$</span> x <span class="math-container">$n^f$</span> decoding output linear matrix? To me, it seems more natural for both these matrices to be learned independently from each other via the training data, right? Or am I misinterpreting the paper?</p>
  
Tags: <natural-language-processing><transformer><bert><attention>
Title: Transformers: how does the decoder final layer output the desired token?
UserDisplayName: user3667125
ThreadText: # Transformers: how does the decoder final layer output the desired token?
  
  (Asked by: username_0 on 2020-12-06)
  
  In the paper [Attention Is All You Need](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf), this section confuses me:
  
  > In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear transformation [output of the decoding section]
  
  Shouldn't the weights be different, and not the same? Here is my understanding:
  
  For simplicity, let us use the English-to-French translation task where we have $n^e$ number of English words in our dictionary and $n^f$ number of French words.
  
  * In the encoding layer, the input tokens are $1$ x $n^e$ one-hot vectors, and are embedded with a $n^e$ x $d^{model}$ learned embedding matrix.
  * In the output of the decoding layer, the final step is a linear transformation with weight matrix $d^{model}$ x $n^f$, and then applying softmax to get the probability of each french word, and choosing the french word with the highest probability.
  
  How is it that the $n^e$ x $n^{model}$ input embedding matrix share the same weights as the $d^{model}$ x $n^f$ decoding output linear matrix? To me, it seems more natural for both these matrices to be learned independently from each other via the training data, right? Or am I misinterpreting the paper?
  
  **Tags:** natural-language-processing, transformer, bert, attention
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2020-12-06. Score: 2 | Accepted answer
  
  I found the answer by reading the paper referenced by that section, [Using the output embedding to improve language models](https://arxiv.org/pdf/1608.05859.pdf)
  
  > Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of
  > the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of this model is the union of both the source
  > and target vocabularies. In this model, both in the
  > encoder and decoder, all subwords are embedded
  > in the same duo-lingual space.
  
  It seems like they learned a single embedding matrix ($n^e + n^f$) x $d^{model}$ in dimension.
  
  
  ---
  
  ### Answer by username_1 on 2023-07-23. Score: 0
  
  Well actually the dictionary does not contain entire words but rather [byte pair encodings](https://en.m.wikipedia.org/wiki/Byte_pair_encoding). At the end of chapter 3 of the [paper](https://arxiv.org/pdf/1608.05859.pdf) that you listed they briefly mention that around 85-90% of the subwords are shared between english and french (see also Table 1). So in this case sharing the weights between the source embedding layer and the target embedding layer makes a lot of sense.
  
  I think that if you are in a different setting (e.g. english to chinese), where your final layer contains $n^e + n^f$ outputs, then this doesn't make much sense because during decoding you actually want to generate words only from the target language, and training this huge softmax layer might be really tricky and hard to do.
  
  Now regarding sharing the weights between the **decoder** embedding layer and the **decoder** output layer. I think this is a very nice idea, and you can always do this regardless of the source language or the task altoghether. However, you might want to be careful with the initialization of these weights. You can checkout [this](https://username_1.github.io/posts/transformer/#token-embedding-layer) if you want to read more.
  
  
  ---
  
  