Is summing of attention_mask intended?
# ❓ Questions & Help
Is summing of `attention_mask` intended?

## Details
Documents describe `attention_mask` as a mask to avoid performing attention on padding token indices.
However from the [code](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/modeling_bert.py#L243) in `BertSelfAttention` the attention mask is added to the scores.
```
attention_scores = attention_scores + attention_mask
```

Is this intended? Does this fully mask out the context?


Yes, it's intended. The attention mask has values of 0 where its attending to the tokens, and has a value of `-10000` for tokens it's not attending. By summing this attention mask, it zeros out the attentions that should not be kept. See [here](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/modeling_utils.py#L228) for the implementation.<|||||>@LysandreJik  I found that `attention_mask` is simply computed by `encoded_inputs["attention_mask"] = [1] * len(encoded_inputs["input_ids"])` in [tokenization_utils_base.py](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/tokenization_utils_base.py#L1944). Should `attention_mask` be synchronized with `encoded_inputs["input_ids"]` when a [MASK] appears in the input?  For example,

Input: This is [MASK] pen.

Should the `attention_mask` be `1 1 0 1` rather than `1 1 1 1`?<|||||>The attention mask is computed according to the padding tokens, not masking tokens. See [here](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/tokenization_utils_base.py#L1932) for a sequence that requires padding.