Answers.Body: <p>No, adding an additional mask to the residual connections or the linear layer isn't necessary. The masking is crucial solely for the attention mechanism. Implementing a mask in the residual connections or the linear layer would only reduce dimensionality and overly constrain the model's outputs. Remember, for the final output, it's essential that the model has full contextual information. Zeroing a value at any step would make the model less efficient by causing it to lose information.</p>
  <p>It's important to note that even within the attention mechanism, the input data is already being utilized in the attention computation. This is evident after the masking of the attention weights, where the matrix <span class="math-container">$QK^T$</span> is projected again with the
  <span class="math-container">$V$</span> matrix, which is equal to <span class="math-container">$W_VX$</span>. Where <span class="math-container">$X$</span> is the input data. Thus, worrying about retaining the mask output is unnecessary. Additionally, when multiplying with the <span class="math-container">$V$</span> matrix, you're already losing the lower triangular matrix. From that point on, adding the initial input doesn't pose an issue.</p>
  <p>Another crucial aspect to consider is the significance of the residual layers. Without them, the positional information from the Positional Encoding (PE) would rapidly diminish, making it vital to retain these layers. So, even if, for some reason, it affects the property of not seeing future tokens (which it doesn't), you would still want to maintain the residual connections.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T16:47:48.343Z
Answers.Text: Regarding "Positional Encoding (PE) would rapidly diminish", will it be happened in this neural network model? This model has only two forward layers not around 20 layers?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T20:47:36.980Z
Answers.Text: In a certain degree it will still happen, and the residual connections in the model evidence that. Another reason for the Residuals is the vanishing gradient effect. They are still needed for solving this issue if it only has 2 forward layers? Probably not as necessary for 20 layers but it can help. Another consideration is that technically speaking very few implementations use the full Transformer architecture. Most language models use BERT or GPT that are stacked multi-head attention layers, so in practice Transformers do use a bunch of layers.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T22:10:27.950Z
Answers.Text: Thanks for aspect of application of Transformer. Is there any reference we could see entire model structure of BERT or GPT? When I search in Google, BERT looks like 12 to 24 transformers from https://www.researchgate.net/figure/The-Transformer-based-BERT-base-architecture-with-twelve-encoder-blocks_fig2_349546860. For me, adding, skipping layer, at 24 layers is still questionable. Have you thought that Masked Attention isn't actually not working?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T00:30:31.180Z
Answers.Text: I understood the implementation of BERT and GPT with these repos nanoGPT and BERT-pytorch. In general, both architectures can be scaled indefinitely, there is no limit to how many transformer blocks you can use, so 24 is really an arbitrary number, it can be any number. For the other question, I did not find it questionable to add the skip connections, it is a very common practice. Then Masked Attention not working would depend on the objective to solve. This helps the model process information with human-like reasoning (because humans can't process language in chunks but as a sequence)
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T09:07:43.470Z
Answers.Text: "...add the skip connections, it is a very common practice" I actually haven't seen any other with the masked layer. All other neural network models seem like not having masking operation. If you find any, please share.
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T05:37:31.460Z
Answers.LastActivityDate: 2023-11-19T05:37:31.460Z
Answers.UserDisplayName: Cesar Ruiz
Body: <p>I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input to the output of masked multi-head attention block adds some of the information from future time steps which is then used to construct the query matrix for multi-head attention block.</p>
  <p>Shouldn't some kind of mask be applied before adding the input values to the output of masked multi-head attention block as well?</p>
  <p><em>Asking this question in the context of training process.</em></p>
  <p><img src="https://kikaben.com/transformers-encoder-decoder/images/figure-1.png" alt="Reference Image for Decoder Architecture" /></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-16T22:57:48.387Z
Comments.Text: What do you think the output of Masked (Multi-Head) Attention, if the input is "I am a boy"?
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-18T09:47:50.057Z
Comments.Text: The output would comprise of 4 vectors, each corresponding to one of the words in input sequence (assuming tokenization is word based). The output vector corresponding to word "am" will only be determined by the embeddings corresponding to "I" and "am", no information about "a" and "boy" is used for its computation. Masking is done by adding upper-triangular matrix with all values equal to $-\infty$ before applying softmax to obtain attention weights. Multi-Head just means that several attention blocks will be used, output from which will be concatenated and mapped back to same dimensions.
Comments.UserDisplayName: SAGALPREET SINGH
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-20T16:45:05.473Z
Comments.Text: If the output of four vectors, could you guess how to add the input to the output (and normalized)?
Comments.UserDisplayName: Cloud Cho
Tags: <natural-language-processing><transformer><decoder>
Title: Masking in Decoder of Transformer
UserDisplayName: SAGALPREET SINGH
ThreadText: # Masking in Decoder of Transformer
  
  (Asked by: username_0 on 2023-11-12)
  
  I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input to the output of masked multi-head attention block adds some of the information from future time steps which is then used to construct the query matrix for multi-head attention block.
  
  Shouldn't some kind of mask be applied before adding the input values to the output of masked multi-head attention block as well?
  
  *Asking this question in the context of training process.*
  
  ![Reference Image for Decoder Architecture](https://kikaben.com/transformers-encoder-decoder/images/figure-1.png)
  
  **Tags:** natural-language-processing, transformer, decoder
  
  > **Comments:**
  > **username_1**: What do you think the output of Masked (Multi-Head) Attention, if the input is "I am a boy"?  
  > **username_0**: The output would comprise of 4 vectors, each corresponding to one of the words in input sequence (assuming tokenization is word based). The output vector corresponding to word "am" will only be determined by the embeddings corresponding to "I" and "am", no information about "a" and "boy" is used for its computation. Masking is done by adding upper-triangular matrix with all values equal to $-\infty$ before applying softmax to obtain attention weights. Multi-Head just means that several attention blocks will be used, output from which will be concatenated and mapped back to same dimensions.  
  > **username_1**: If the output of four vectors, could you guess how to add the input to the output (and normalized)?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-19. Score: 0
  
  No, adding an additional mask to the residual connections or the linear layer isn't necessary. The masking is crucial solely for the attention mechanism. Implementing a mask in the residual connections or the linear layer would only reduce dimensionality and overly constrain the model's outputs. Remember, for the final output, it's essential that the model has full contextual information. Zeroing a value at any step would make the model less efficient by causing it to lose information.
  
  It's important to note that even within the attention mechanism, the input data is already being utilized in the attention computation. This is evident after the masking of the attention weights, where the matrix $QK^T$ is projected again with the
  $V$ matrix, which is equal to $W\_VX$. Where $X$ is the input data. Thus, worrying about retaining the mask output is unnecessary. Additionally, when multiplying with the $V$ matrix, you're already losing the lower triangular matrix. From that point on, adding the initial input doesn't pose an issue.
  
  Another crucial aspect to consider is the significance of the residual layers. Without them, the positional information from the Positional Encoding (PE) would rapidly diminish, making it vital to retain these layers. So, even if, for some reason, it affects the property of not seeing future tokens (which it doesn't), you would still want to maintain the residual connections.
  
  > **Comments:**
  > **username_1**: Regarding "Positional Encoding (PE) would rapidly diminish", will it be happened in this neural network model? This model has only two forward layers not around 20 layers?  
  > **username_2**: In a certain degree it will still happen, and the residual connections in the model evidence that. Another reason for the Residuals is the vanishing gradient effect. They are still needed for solving this issue if it only has 2 forward layers? Probably not as necessary for 20 layers but it can help. Another consideration is that technically speaking very few implementations use the full Transformer architecture. Most language models use BERT or GPT that are stacked multi-head attention layers, so in practice Transformers do use a bunch of layers.  
  > **username_1**: Thanks for aspect of application of Transformer. Is there any reference we could see entire model structure of BERT or GPT? When I search in Google, BERT looks like 12 to 24 transformers from https://www.researchgate.net/figure/The-Transformer-based-BERT-base-architecture-with-twelve-encoder-blocks_fig2_349546860. For me, adding, skipping layer, at 24 layers is still questionable. Have you thought that Masked Attention isn't actually not working?  
  > **username_2**: I understood the implementation of BERT and GPT with these repos nanoGPT and BERT-pytorch. In general, both architectures can be scaled indefinitely, there is no limit to how many transformer blocks you can use, so 24 is really an arbitrary number, it can be any number. For the other question, I did not find it questionable to add the skip connections, it is a very common practice. Then Masked Attention not working would depend on the objective to solve. This helps the model process information with human-like reasoning (because humans can't process language in chunks but as a sequence)  
  > **username_1**: "...add the skip connections, it is a very common practice" I actually haven't seen any other with the masked layer. All other neural network models seem like not having masking operation. If you find any, please share.  
  
  ---
  
  