is_index_masked and is_index_global_attn in Longformer
I noticed that in [LongformerEncoder](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L1255), is_index_masked and is_index_global_attn is set like this:

```
is_index_masked = attention_mask < 0
is_index_global_attn = attention_mask > 0
```

Can anyone help me understand what does these two variables do and why they are set in this way?

Note that in [LongformerModel](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L1599), we have final attention_mask => 0 for no attention, 1 for local attention 2 for global attention.

Which means in LongformerEncoder, we will get is_index_masked as all False, and is_index_global_attn as all True except for no attention token. This is quite confusing to me.

It looks to me that the correct value should be like this, though I could be wrong:

```
is_index_masked = attention_mask == 0
is_index_global_attn = attention_mask == 2
```

Any help is appreciated!
cc @patrickvonplaten <|||||>Hey @lightxu,

What we usually do in `transformers` is to convert an attention_mask which looks like:

```python
attention_mask = [[1, 1, 1, 1], [1, 0, 0, 0]]
```

to something like:

```python
attention_mask = torch.broadcast_to([[0.0, 0.0, 0.0, 0.0], [0, -10000.0, -10000,0, -10000,0]], (1, 1, 4, 4))
```

since a `1` means that the vector at this time step of the `QK^T` matrix is **not** masked (+ 0.0) and `0` means that it is masked (+-10000.0). This happens for all models and is standard practice for transformer models. 

Now Longformer is special because it requires2 types of attention mechanism inside meaning that in addition to `0.0` and `-10000.0` we also add `10000.0` which is why we have the statement:

```python
is_index_masked = attention_mask < 0
is_index_global_attn = attention_mask > 0
```

Does that make sense?<|||||>Hi @patrickvonplaten 

This does make sense, since I also see corresponding comments in [LongformerSelfAttention](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L561). I am trying to understand where does this conversion happen, is it happenining here in [LongformerModel](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_longformer.py#L1699)?

For the context, I am trying to adapt distilbert model to use longformer self attention. The structure of [distilbert model](https://github.com/huggingface/transformers/blob/v4.3.0/src/transformers/models/distilbert/modeling_distilbert.py#L474) looks pretty simple, and I couldn't find anything that broadcast the attention_mask.<|||||>Found the conversion in [`get_extended_attention_mask`](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L311) method. So I think the logic in Longformer is correct, I will try to make similar changes for my long version of distilbert model. Thanks!