Hi.
When we don’t pass decoder_attention_mask to BartModel, the model automatically creates decoder input masks with _make_causal_mask.
I’ve noticed that the method inserts ‘0’ in mask positions corresponding to indices the model needs to attend, and -inf in positions corresponding to indices to be ignored. Below is the link to aforementioned code:

  

      github.com
  

  
    huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/src/transformers/models/bart/modeling_bart.py#L85



      
      
              return shifted_input_ids
          
          

          
def _make_causal_mask(input_ids_shape: torch.Size, dtype: torch.dtype, past_key_values_length: int = 0):
              """
              Make causal mask used for bi-directional self-attention.
              """
              bsz, tgt_len = input_ids_shape
              mask = torch.full((tgt_len, tgt_len), float("-inf"))
              mask_cond = torch.arange(mask.size(-1))
              mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
              mask = mask.to(dtype)
          
          
    if past_key_values_length > 0:
                  mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype), mask], dim=-1)
              return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
          
          

          
def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
              """
              Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
      
    
  


  

  
    
    
  

  


As far as I know attention masks should have 1 in indices we want to attend. Could anyone shed some light on this?