Last hidden states different for same input even when evaluating ViT model
### System Info

Google Colab

### Who can help?

@NielsRogge @sg

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've created a public Colab notebook: https://colab.research.google.com/drive/1CUpyNInQg2kw7gL-mYwX8ov8fc9HuAV-#scrollTo=DAXcuQXVtTkz

where a VitMAE model is created, and its weights saved using Trainer.

Then it is loaded again, and placed in eval mode. However the last hidden state changes when it is passed the same input - why might that be happening?

### Expected behavior

A model in eval mode should have a deterministic output for the same input
Hi,

Yes ViTMAE generates a random boolean mask to mask patches internally. This results in non-deterministic hidden states when you perform a forward pass twice on the same image.

To get deterministic behaviour, you can pass a noise tensor yourself as shown [here](https://github.com/huggingface/transformers/blob/afce73bd9d891b55dcb8d4d875d17718ffa01ff0/tests/models/vit_mae/test_modeling_vit_mae.py#L321). <|||||>Thanks!<|||||>Btw another way (in case you don't want to mask patches) is to set `config.mask_ratio=0.0`