MAE Noise Was NOT Removed In Prediction For TFViTMAEModel / ViTMAEModel
### System Info

transformers version: v4.24.0 (latest)


### Who can help?

@NielsRogge 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import numpy as np
from transformers import AutoFeatureExtractor, TFViTMAEModel
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-mae-base")
model = TFViTMAEModel.from_pretrained("facebook/vit-mae-base")
inputs = feature_extractor(images=image, return_tensors="tf")
outputs1 = model(**inputs)
outputs2 = model(**inputs)
res1 = outputs1.last_hidden_state.numpy()
res2 = outputs2.last_hidden_state.numpy()
np.allclose(res1, res2, rtol=1e-04, atol=1e-05)  # This returns False as noises are introduced.
```

### Expected behavior

It should return True.

It returns False as noises are added in `random_masking()` which hasn't been removed for prediction. There are no options to do that.
Hi,

ViTMAE generates a random mask internally to mask out patches. Hence forwarding the same inputs twice through the model will result in different hidden states. If you want to ensure reproducability, you can pass the `noise` argument. See [here](https://github.com/huggingface/transformers/blob/8503cc755050c6ed5bc771e3244c29b71be1841e/tests/models/vit_mae/test_modeling_vit_mae.py#L317) for an example.<|||||>Thanks @NielsRogge for the reply. I'm using this model to extract image features and I'm using some hacks to remove noise and masks for inference.

After reading the [ViTMAE code](https://github.com/facebookresearch/mae/blob/main/FINETUNE.md), I found that the trained ViTMAE model can be loaded as a regular ViT model directly.  Just tested with `TFViTMAEModel.from_pretrained("facebook/vit-mae-base")` and it works. Could you help confirm is this the correct approach to use pre-trained MAE model? <|||||>Yes, that's also shown in the docs. Note that `TFViTMAEModel.from_pretrained("facebook/vit-mae-base")` will just load the base Encoder, which does not include the decoder used during pre-training. For that you would need to load `TFViTMAEForPreTraining`.

However, since you want to use the model for image features, it's indeed adviced to just load the Transformer encoder.