Further investigation shows this behavior is desired since attention mask is added to attention weights, so that 0 attention mask value preserves the inputs while -inf attention mask value “masks out” the inputs. (related code pasted below)
However, shouldn’t the encoder attention mask be initialized the same way (0 for relevant inputs, -inf for padding inputs) as well?
Currently, the documentation says encoder attention mask values should be 1 for relevant inputs and 0 for padding inputs.

  

      github.com
  

  
    huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/src/transformers/models/bart/modeling_bart.py#L223



      
      
          if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
              raise ValueError(
                  f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is {attn_weights.size()}"
              )
          
          
if attention_mask is not None:
              if attention_mask.size() != (bsz, 1, tgt_len, src_len):
                  raise ValueError(
                      f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}"
                  )
              attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
              attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
          
          
attn_weights = F.softmax(attn_weights, dim=-1)
          
          
if layer_head_mask is not None:
              if layer_head_mask.size() != (self.num_heads,):
                  raise ValueError(
                      f"Head mask for a single layer should be of size {(self.num_heads,)}, but is {layer_head_mask.size()}"
                  )
              attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)