Answers.Body: <p>The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training.</p>
  <p>This is reflected in <a href="https://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="nofollow noreferrer">The Annotated Transformer</a> as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder:</p>
  <pre><code>class Decoder(nn.Module):
      &quot;Generic N layer decoder with masking.&quot;
      def __init__(self, layer, N):
          super(Decoder, self).__init__()
          self.layers = clones(layer, N)
          self.norm = LayerNorm(layer.size)
          
      def forward(self, x, memory, src_mask, tgt_mask):
          for layer in self.layers:
              x = layer(x, memory, src_mask, tgt_mask)
          return self.norm(x)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T00:10:26.617Z
Answers.LastActivityDate: 2020-12-08T00:10:26.617Z
Answers.UserDisplayName: user3667125
Body: <p>The Decoder mask, also called &quot;look-ahead mask&quot;, is applied in the Decoder side to prevent it from attending future tokens. Something like this:</p>
  <pre><code>[0, 1, 1, 1, 1]
  [0, 0, 1, 1, 1]
  [0, 0, 0, 1, 1]
  [0, 0, 0, 0, 1]
  [0, 0, 0, 0, 0]
  </code></pre>
  <p>But is this mask applied only in the first Decoder block? Or to all its blocks?</p>
  
Tags: <transformer><attention>
Title: Is the Decoder mask (triangular mask) applied only in the first decoder block, or to all blocks in Decoder?
UserDisplayName: Leevo
ThreadText: # Is the Decoder mask (triangular mask) applied only in the first decoder block, or to all blocks in Decoder?
  
  (Asked by: username_0 on 2020-12-07)
  
  The Decoder mask, also called "look-ahead mask", is applied in the Decoder side to prevent it from attending future tokens. Something like this:
  
  ```
  [0, 1, 1, 1, 1]
  [0, 0, 1, 1, 1]
  [0, 0, 0, 1, 1]
  [0, 0, 0, 0, 1]
  [0, 0, 0, 0, 0]
  
  ```
  
  But is this mask applied only in the first Decoder block? Or to all its blocks?
  
  **Tags:** transformer, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-08. Score: 1 | Accepted answer
  
  The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training.
  
  This is reflected in [The Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html) as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder:
  
  ```
  class Decoder(nn.Module):
      "Generic N layer decoder with masking."
      def __init__(self, layer, N):
          super(Decoder, self).__init__()
          self.layers = clones(layer, N)
          self.norm = LayerNorm(layer.size)
          
      def forward(self, x, memory, src_mask, tgt_mask):
          for layer in self.layers:
              x = layer(x, memory, src_mask, tgt_mask)
          return self.norm(x)
  
  ```
  
  
  ---
  
  