CLIPTextModel gives invalid output for zeroed attention mask
### System Info

- `transformers` version: 4.21.2
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.33
- Python version: 3.8.6
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.12.0+cu102 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.2 (cpu)
- Jax version: 0.3.10
- JaxLib version: 0.3.10
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@patil-suraj

### Reproduction

```python
from transformers import CLIPTextModel
import torch

model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32")

inputs = {
  "input_ids": torch.tensor([[49406, 320, 1125, 539, 320, 1929, 49407]]),
  "attention_mask": torch.tensor([[0, 0, 0, 0, 0, 0, 0]])
}

outputs = model(**inputs)
```

Given the zeroed attention mask, the attention weights should be all equal here:

https://github.com/huggingface/transformers/blob/21f6f58721dd9154357576be6de54eefef1f1818/src/transformers/models/clip/modeling_clip.py#L246

However, causal and attention masks are added separately ([here](https://github.com/huggingface/transformers/blob/21f6f58721dd9154357576be6de54eefef1f1818/src/transformers/models/clip/modeling_clip.py#L228-L246)), so in this case, before going through softmax, certain values are twice as small as the other ones (to be more precise, some values are -min_float and other are -inf). Consequently, softmax outputs probabilities that match the causal mask.

This is also the case for `TFCLIPTextModel`.
I've also experienced this issue, and anecdotally this inconsistency seems to impact the quality of stable diffusion outputs from https://github.com/huggingface/diffusers. More specifically, when trying to port the stable diffusion pipeline to another framework (e.g. like Flax where the implementation is not present) using the provided stable diffusion weights, the images which rely on PT's CLIPTextModel are mostly incoherent.

I am assuming Stable Diffusion was trained using the PT CLIPTextModel, and thus results rely on this inconsistent/invalid text embedding? <|||||>Thanks a lot for the issue @jonatanklosko !

This indeed seems a bit strange, I see two solutions here

- We could combine causal mask and attention mask and see what happens
- Or instead of using additive masks, we could replace the masked values with large negative numbers.

however, is it really a bug ? as long as the values for masked positions are much lower than non-masked tokens, those tokens will still be ignored. Do you have an example where you see the masked positions are not ignored ?

@seanmor5 good point! 
>I've also experienced this issue, and anecdotally this inconsistency seems to impact the quality of stable diffusion outputs from https://github.com/huggingface/diffusers

I'm not sure if this impacts the quality of diffusers, for example, as discussed in this [issue](https://github.com/huggingface/diffusers/issues/233), we have verified that the results are 1:1 with the original repo.

>I am assuming Stable Diffusion was trained using the PT CLIPTextModel, and thus results rely on this inconsistent/invalid text embedding?

Yes, it was trained using `CLIPTextModel`, but both this training and the actual pre-trained CLIP model never used attention mask. They always pad the sequence to max_len 77 and use causal mask. This is how we recommend to use the stable diffusion model. I know this is not ideal but that's how it was trained. cf https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/encoders/modules.py#L155

Also cc @patrickvonplaten , wdyt ?<|||||>>e.g. like Flax where the implementation is not present

It's coming soon https://github.com/patil-suraj/stable-diffusion-jax/<|||||>Also quite interested in the use case of masking all text tokens - when would this make sense?<|||||>@patil-suraj consider the following weights

```python
import torch
from torch.nn.functional import softmax

softmax(torch.tensor([1.0, 0.0]))
#=> tensor([0.7311, 0.2689])
```

if we add large negative values, then the actual weights are neglectable and softmax outputs equal values:

```python
softmax(torch.tensor([1.0 - 1e30, 0.0 - 1e30]))
#=> tensor([0.5000, 0.5000])
```

however if we also add large negative values from causal mask this makes the values disproportional:

```python
softmax(torch.tensor([1.0 - 1e30, 0.0 - 1e30 - 1e30]))
#=> tensor([1., 0.])
```

In the flax implementation masks are combined and negative values are added just once (though apparently the output still differs, because the flax version uses `-1e4` as the negative value). But thinking about this, shouldn't all weights be 0 when masking all tokens? If so, modifying the input to softmax is not enough.

@patrickvonplaten

> Also quite interested in the use case of masking all text tokens - when would this make sense?

This came up with unconditional input to stable diffusers, I believe [this part](https://github.com/huggingface/diffusers/blob/16172c1c7ef6ec721bfe4d0787313519157749a1/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L92-L94). @seanmor5 please correct me if I'm wrong :)<|||||>Interesting! Thanks for the pointer @jonatanklosko - I think though that even when doing:

The tokenizer should return tokens that should be attended to. E.g. taking the line here: https://github.com/huggingface/diffusers/blob/16172c1c7ef6ec721bfe4d0787313519157749a1/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L92-L94

and doing:

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("openai/clip-vit-large-patch14")

uncond_input = tok(
    [""] * 2, padding="max_length", max_length=77, return_tensors="pt"
)
print(uncond_input)
```
I get:

```
{'input_ids': tensor([[49406, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407],
        [49406, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407]]), 'attention_mask': tensor([[1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0],
        [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0]])}
```

which shows that at least two tokens are attended to (so > 0 positions of attention_mask are equal to 1) which should avoid the problem described above. What would be the use case where all values in `attention_mask` are 0?<|||||>@patrickvonplaten Ahh! This was my mistake, when doing some initial testing with the stable diffusion weights in our framework I noticed after a step or 2 the latent results started to diverge from the PT stable diffusion implementation. I noticed some slight differences between the results of the text encoder from PT and our implementation (within some reasonable amount of precision) and ended up writing an invalid test case where all of the attention masks were 0. The source of our issue probably lies elsewhere then, sorry for the confusion!<|||||>No worries! Better be safe than sorry! :-) <|||||>So I think we can close this :)

The additive mask before softmax is a trick that works under the assumption that the mask has at least a single 1. So from what I understand can be said that for zeroed mask the output of most models is just not well defined, and it's fine given the use cases so far.