Add padding changes the output of BertModel
### System Info

- `transformers` version: 4.30.1
- Platform: Linux-4.4.0-131-generic-x86_64-with-glibc2.10
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import matplotlib.pyplot as plt
import numpy as np
import torch
from transformers import BertModel, __version__

print(f"torch version: {torch.__version__}")
print(f"transformers version: {__version__}")

np.random.seed(42)
torch.manual_seed(42)

torch.set_default_dtype(torch.float32)

# Load model
model = BertModel.from_pretrained('bert-base-uncased')
model = model.eval()

# pad and mask
x = [10]
m = [1]
outs = []
for pads in range(511):
    xt = torch.LongTensor([x])
    mt = torch.FloatTensor([m])
    out = model(xt, attention_mask=mt).last_hidden_state[0, 0, 0].item()
    outs.append(out)
    x = x + [0]
    m = m + [0]

# plot outs
outs = np.array(outs)
outs = outs - outs[0]
plt.figure(figsize=(4, 4), dpi=80)
plt.plot(outs)
plt.show()
```

### Expected behavior

```
torch version: 2.0.1+cu117
transformers version: 4.30.1
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```
![test](https://github.com/huggingface/transformers/assets/32625090/cc05b73c-8ed6-42d5-ac58-7a8c51e171ad)

(x: pad length; y: error)

And `torch.set_default_dtype(torch.float64)` reduces this error to ~1e-15.
Hi @AaronNing, thanks for opening this issue! 

The (very small!) differences are arising because of the forced typing with `torch.set_default_dtype(torch.float32)`. Floating point arithmetic is inherently imprecise. By forcing tensors to be `float32`, where previously they weren't, you're introducing imprecision into architecture and ultimately the outputs. As you've observed, the difference is v. small if this is changed to `float64`. This is because the model can use more memory to more accurately represent numbers and perform calculations. 

Below are the results if this type casting doesn't occur. As you can see, there is now no difference when introducing padding:
![floating_point_arithmetic](https://github.com/huggingface/transformers/assets/22614925/e639bec0-2870-4008-bf35-c1a31584fc25)

For reference, when we're porting models into our library, we consider absolute differences on the order of ~1e-6 to be acceptable. <|||||>@amyeroberts Thanks for your reply! 
However, I removed the `torch.set_default_dtype(torch.float32)` line (with others unchanged) from the code above but still got the same figure. Did you modify other parts of the code? Thanks.

Also, according to [PyTorch's official document](https://pytorch.org/docs/stable/generated/torch.set_default_dtype.html), 
> When PyTorch is initialized its default floating point dtype is torch.float32

So I suspect that this operation is not what caused the inaccuracy?<|||||>@AaronNing  - you're right, my bad, I misinterpreted what `torch.set_default_dtype` was doing. Removing that was the only change I made to the code, however, like you, adding it back didn't affect behaviour and I observed the same results (no change with padding). 

In this case, I suspect the differences might be due to hardware, which can affect float computations, I'm running on CPU with Mac M1. Other than that, I don't have a good guess.  <|||||>> @AaronNing - you're right, my bad, I misinterpreted what `torch.set_default_dtype` was doing. Removing that was the only change I made to the code, however, like you, adding it back didn't affect behaviour and I observed the same results (no change with padding).
> 
> In this case, I suspect the differences might be due to hardware, which can affect float computations, I'm running on CPU with Mac M1. Other than that, I don't have a good guess.

I see, thanks. BTW do you have any idea why (in your plot) the output is different when input length = 1?<|||||>> BTW do you have any idea why (in your plot) the output is different when input length = 1?

I think this is just because the diff is calculated as: 

```python
outs = outs - outs[0]
```

And the first element with have ~0 difference with itself. <|||||>@amyeroberts @AaronNing 

Hi, I also met the same issue. I find that it may due to layernorm operation, since different sequence length will lead to different norm results.

To verify this, you can print the results [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L239).

And I tested it is hardware independent and not related to python accuracy error. I believe it could be a common issue for all transformer-based models.<|||||>@StevenTang1998 Could you share which different hardware you ran this on? Were the differences you saw from running the same script that @AaronNing provided? If not, could you share yours? <|||||>Yes, I have the similar results using the @AaronNing 's code.

![1](https://github.com/huggingface/transformers/assets/37647985/b190d53d-8bf0-4750-bd83-06d5e72fe7cd)

I find that it may due to layernorm operation, since different sequence length will lead to different norm results.
To verify this, you can print the results [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L239).

This the system info of mine:
- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.13.1 (True)<|||||>Hi @StevenTang1998, with respect to the hardware, what I meant was which hardware have you run this on to confirm that it's invariant? 

If I run on my CPU I get the same as before - ~1e-7 difference across all sequence lengths

If I run on GPU, I see mostly ~ 1e-6 difference

![image](https://github.com/huggingface/transformers/assets/22614925/f005bf40-a018-47f9-8948-82978bb1f788)

For my own experiments, the difference in observations seems to be arising from the hardware. <|||||>I conducted the experiment on RTX 3090 GPUs.
<|||||>@StevenTang1998 In order to confirm it's invariant one must run on at least two different pieces of hardware - ideally CPU and GPU - and obtain the same results.<|||||>@amyeroberts 

I reran the experiments and want to reclaim that I don't believe it is an coincidence since @AaronNing and I both got the variant results. 
**I have printed the results before and after the layernorm. The results before the layernorm are exactly the same regardless the pad length, while the results after the layernorm get variant.**
Since the layernorm will normalize all the word embeddings in one sequence, I think add pad tokens will affect the normalization results.

- CPU
![cpu](https://github.com/huggingface/transformers/assets/37647985/b35304d5-25c7-4e41-9884-58cb72ba40ef)

- GPU (3090 and A100 have the same results)
![a100](https://github.com/huggingface/transformers/assets/37647985/52b9055c-9e74-4822-b2ce-763f2117b926)

<|||||>@StevenTang1998 Thanks. It sounds reasonable that LN caused the difference. 
> I have printed the results before and after the layernorm. The results before the layernorm are exactly the same regardless the pad length, while the results after the layernorm get variant.

It would be very nice if you provide one or two figures to demonstrate that, though I'm already taking it. 
> I believe it could be a common issue for all transformer-based models.

I agree. Maybe when the model is well-trained, this difference will be reasonably small (as shown in our experiments), so this is not regarded as a big issue.