Answers.Body: <p>I have read the OpenNMT source code (<a href="https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py" rel="nofollow noreferrer">https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py</a>).</p>
  <p>It seems like an extra linear layer learns the weights <span class="math-container">$W^{key}$</span> and <span class="math-container">$W^{value}$</span> (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by <span class="math-container">$W^{key}$</span> to get the keys, and by <span class="math-container">$W^{value}$</span> to get the values.</p>
  <p>Additionally, these weights and biases seem to be independent across each of the decoding layer. So you feed the same encoder output (add + norm layer output), but multiply by different <span class="math-container">$W^{key}$</span> and <span class="math-container">$W^{value}$</span> matrices and add by different biases for each of the decoding layer, resulting in different keys and values for each layer</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-06T22:29:03.577Z
Answers.LastActivityDate: 2020-12-06T22:29:03.577Z
Answers.UserDisplayName: user3667125
Body: <p>I was reading the paper <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="nofollow noreferrer">Attention Is All You Need</a>.</p>
  <p>It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a <span class="math-container">$n$</span> x <span class="math-container">$d^{model}$</span> matrix, where <span class="math-container">$n$</span> is the length of the input to the encoder.</p>
  <p>How do we convert this <span class="math-container">$n$</span> x <span class="math-container">$d^{model}$</span> matrix into the keys <span class="math-container">$K$</span> and values <span class="math-container">$V$</span> that are fed into the decoder's encoder-decoder attention step?</p>
  <p>Note that, if <span class="math-container">$h$</span> is the number of attention heads in the model, the dimensions of <span class="math-container">$K$</span> and <span class="math-container">$V$</span> should both be <span class="math-container">$n$</span> x <span class="math-container">$\frac{d^{model}}{h}$</span>. For <span class="math-container">$h=8$</span>, this means we need a <span class="math-container">$n$</span> x <span class="math-container">$\frac{d^{model}}{4}$</span> matrix.</p>
  <p>Do we simply add an extra linear layer that learns a <span class="math-container">$d^{model}$</span> x <span class="math-container">$\frac{d^{model}}{4}$</span> weight matrix?</p>
  <p>Or do we use the output of the final Add &amp; Norm layer, and simply use the first <span class="math-container">$\frac{d^{model}}{4}$</span> columns of the matrix and discard the rest?</p>
  
Tags: <natural-language-processing><transformer><bert><attention>
Title: Transformers: how to get the output (keys and values) of the encoder?
UserDisplayName: user3667125
ThreadText: # Transformers: how to get the output (keys and values) of the encoder?
  
  (Asked by: username_0 on 2020-12-06)
  
  I was reading the paper [Attention Is All You Need](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf).
  
  It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ matrix, where $n$ is the length of the input to the encoder.
  
  How do we convert this $n$ x $d^{model}$ matrix into the keys $K$ and values $V$ that are fed into the decoder's encoder-decoder attention step?
  
  Note that, if $h$ is the number of attention heads in the model, the dimensions of $K$ and $V$ should both be $n$ x $\frac{d^{model}}{h}$. For $h=8$, this means we need a $n$ x $\frac{d^{model}}{4}$ matrix.
  
  Do we simply add an extra linear layer that learns a $d^{model}$ x $\frac{d^{model}}{4}$ weight matrix?
  
  Or do we use the output of the final Add & Norm layer, and simply use the first $\frac{d^{model}}{4}$ columns of the matrix and discard the rest?
  
  **Tags:** natural-language-processing, transformer, bert, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-12-06. Score: 3 | Accepted answer
  
  I have read the OpenNMT source code (<https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py>).
  
  It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by $W^{key}$ to get the keys, and by $W^{value}$ to get the values.
  
  Additionally, these weights and biases seem to be independent across each of the decoding layer. So you feed the same encoder output (add + norm layer output), but multiply by different $W^{key}$ and $W^{value}$ matrices and add by different biases for each of the decoding layer, resulting in different keys and values for each layer
  
  
  ---
  
  