Questions on ViT and ViT-MAE model image preprocessing
### System Info

Latest version

### Who can help?

@NielsRogge

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi @NielsRogge, I'm reading the ViT image processing from HF and its original implementation and have a few questions on image processing. Really appreciate your helping me understand the differences.

**ViT**
The JAX implementation from Google rescales the pixel range to [-1, 1]: https://github.com/google-research/vision_transformer/blob/main/vit_jax/input_pipeline.py#L214
HF rescales it by a factor of 1/255: https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/image_processing_vit.py#L80

**ViT MAE**
PyTorch implementation from Meta resizes the image with PIL.Image.BICUBIC interpolation: https://github.com/facebookresearch/mae/blob/efb2a8062c206524e35e47d04501ed4f544c0ae8/util/datasets.py#L59

HF uses BILINEAR https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/image_processing_vit.py#L78

### Expected behavior

I'd like to have clarifications of the questions above.
Hi,

Thanks for your interest in ViT and ViT MAE!

Regarding point 1; ViT was ported from the timm library, which also uses a scale factor of 1/255. This can be verified as follows:

```
from timm.data import resolve_data_config
from timm.data.transforms_factory import create_transform

model = timm.create_model('vit_base_patch16_224',pretrained=True)
model.eval()

# Create Transform
transform = create_transform(**resolve_data_config(model.pretrained_cfg, model=model))
print(transform)
```
which prints:
```
Compose(
    Resize(size=248, interpolation=bicubic, max_size=None, antialias=None)
    CenterCrop(size=(224, 224))
    ToTensor()
    Normalize(mean=tensor([0.5000, 0.5000, 0.5000]), std=tensor([0.5000, 0.5000, 0.5000]))
)
```
=> here, [ToTensor](https://pytorch.org/vision/stable/generated/torchvision.transforms.ToTensor.html) is used, which as stated in the docs converts a tensor with values 0 - 255 to the range [0-1]. Maybe @rwightman can clarify this.

Regarding point 2:

That's a valid point, not sure why I missed that. We could update this, although I'm not sure it has a big impact on downstream results. cc @sgugger <|||||>@NielsRogge @zhoutong-fu

ToTensor: uint8 [0, 255] -> float [0, 1.0]
Normalize(mean=tensor([0.5000, 0.5000, 0.5000]), std=tensor([0.5000, 0.5000, 0.5000])): float [0, 1.0] -> float [-1, 1]

Interpolation should be bicubic for both ViT and ViT MAE


<|||||>Thank you guys for the discussion.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.