The way the model logic is implemented it doesn’t matter.
Specifically because of this function which says only look at word pieces, not special tokens:

  
      github.com
  
  
    huggingface/transformers/blob/edfd82f5ff179f7600d8f2eea204e21bd07d99e4/src/transformers/modeling_utils.py#L192

    else:
        raise ValueError(
            "{} not recognized. `dtype` should be set to either `torch.float32` or `torch.float16`".format(
                self.dtype
            )
        )

    return encoder_extended_attention_mask

def get_extended_attention_mask(self, attention_mask: Tensor, input_shape: Tuple, device: device) -> Tensor:
    """Makes broadcastable attention mask and causal mask so that future and maked tokens are ignored.

    Arguments:
        attention_mask: torch.Tensor with 1 indicating tokens to ATTEND to
        input_shape: tuple, shape of input_ids
        device: torch.Device, usually self.device

    Returns:
        torch.Tensor with dtype of attention_mask.dtype
    """
    # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]



  
  
    
    
  
  


and the output is incorporated here which tells the model what to look at:

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L263


query_layer = self.transpose_for_scores(mixed_query_layer)
key_layer = self.transpose_for_scores(mixed_key_layer)
value_layer = self.transpose_for_scores(mixed_value_layer)

# Take the dot product between "query" and "key" to get the raw attention scores.
attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
attention_scores = attention_scores / math.sqrt(self.attention_head_size)
if attention_mask is not None:
    # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
    attention_scores = attention_scores + attention_mask

# Normalize the attention scores to probabilities.
attention_probs = nn.Softmax(dim=-1)(attention_scores)

# This is actually dropping out entire tokens to attend to, which might
# seem a bit unusual, but is taken from the original Transformer paper.
attention_probs = self.dropout(attention_probs)

# Mask heads if we want to
if head_mask is not None:



  
  
    
    
  
  


If it’s unclear. Put a breakpoint on those lines and run this unit test: https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L521