Answers.Body: <ul>
  <li>the z’s are stacked and then passed to a single feed forward layer</li>
  <li>the difference in shapes of inputs is resolved by padding to the dimension of the model</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T15:16:24.137Z
Answers.Text: Hi @Arjun Reddy! If possible, please add some supporting information to this answer to help other readers understand why the answer is correct. Thanks!
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-06T15:55:00.920Z
Answers.LastActivityDate: 2023-05-06T15:55:00.920Z
Answers.UserDisplayName: Arjun Reddy
Body: <p>Considering the architecture of encoder and decoder in transformer as shown below:
  <a href="https://i.sstatic.net/4pAzL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4pAzL.png" alt="enter image description here" /></a></p>
  <ul>
  <li><strong>Does each input token after self attention mechanism (z1,z2,z3,...)is passed to it's specific  separate Feed forward neural network or does all the Z's are stacked into one and then passed to single FFNN?</strong></li>
  <li><strong>If all the Z's are stacked into one, then how the difference in shapes of different inputs is taken care</strong></li>
  <li><strong>If every z has its own Feed forward neural network, how in practical it is implemented with arbitrary input length?</strong></li>
  </ul>
  
Tags: <transformer><architecture>
Title: Does Number of Fully connected neural networks changes in transformer architechture based on max length input size?
UserDisplayName: Arjun Reddy
ThreadText: # Does Number of Fully connected neural networks changes in transformer architechture based on max length input size?
  
  (Asked by: username_0 on 2023-04-23)
  
  Considering the architecture of encoder and decoder in transformer as shown below:
  [![enter image description here](https://i.sstatic.net/4pAzL.png)](https://i.sstatic.net/4pAzL.png)
  
  * **Does each input token after self attention mechanism (z1,z2,z3,...)is passed to it's specific separate Feed forward neural network or does all the Z's are stacked into one and then passed to single FFNN?**
  * **If all the Z's are stacked into one, then how the difference in shapes of different inputs is taken care**
  * **If every z has its own Feed forward neural network, how in practical it is implemented with arbitrary input length?**
  
  **Tags:** transformer, architecture
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-05-06. Score: 0
  
  * the z’s are stacked and then passed to a single feed forward layer
  * the difference in shapes of inputs is resolved by padding to the dimension of the model
  
  > **Comments:**
  > **username_1**: Hi @Arjun Reddy! If possible, please add some supporting information to this answer to help other readers understand why the answer is correct. Thanks!  
  
  ---
  
  