Answers.Body: <p>One encoder block of the transformer takes as input one tensor X and multiplies that by <span class="math-container">$W_Q$</span>, <span class="math-container">$W_K$</span>, <span class="math-container">$W_V$</span> to calculate <span class="math-container">$Q$</span>, <span class="math-container">$K$</span>, <span class="math-container">$V$</span> needed in self-attention.</p>
  <p>After performing attention and feed-forward this one encoder block returns a single <span class="math-container">$X'$</span> ready to be taken as input for the next encoder block.</p>
  <p>I find this specific post really helpful:</p>
  <p><a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">https://jalammar.github.io/illustrated-transformer/</a></p>
  <p>You can find there this image, which shows that from one single input X, we calculate <span class="math-container">$Q, K, V$</span>. Learning those weight matrices <span class="math-container">$W_Q$</span>, <span class="math-container">$W_K$</span>, <span class="math-container">$W_V$</span> is part of the training of an encoder.
  <a href="https://i.sstatic.net/s6kuJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/s6kuJ.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T22:32:43.787Z
Answers.LastActivityDate: 2023-02-28T22:32:43.787Z
Answers.UserDisplayName: Jakub Podolak
Body: <p>An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1</p>
  <p>How do you stack those ?</p>
  <p>Is there more detailed diagram ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-28T10:26:37.133Z
Comments.Text: Pass the output through 3 different dense layers in parallel, I think.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <transformer><encoder-decoder>
Title: Transformers: how does stacking work?
UserDisplayName: sten
ThreadText: # Transformers: how does stacking work?
  
  (Asked by: username_0 on 2023-02-28)
  
  An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1
  
  How do you stack those ?
  
  Is there more detailed diagram ?
  
  **Tags:** transformer, encoder-decoder
  > **Comments:**
  > **username_1**: Pass the output through 3 different dense layers in parallel, I think.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-28. Score: 2 | Accepted answer
  
  One encoder block of the transformer takes as input one tensor X and multiplies that by $W\_Q$, $W\_K$, $W\_V$ to calculate $Q$, $K$, $V$ needed in self-attention.
  
  After performing attention and feed-forward this one encoder block returns a single $X'$ ready to be taken as input for the next encoder block.
  
  I find this specific post really helpful:
  
  <https://jalammar.github.io/illustrated-transformer/>
  
  You can find there this image, which shows that from one single input X, we calculate $Q, K, V$. Learning those weight matrices $W\_Q$, $W\_K$, $W\_V$ is part of the training of an encoder.
  [![enter image description here](https://i.sstatic.net/s6kuJ.png)](https://i.sstatic.net/s6kuJ.png)
  
  
  ---
  
  