VideoMAE with `num_channels!=3` needs a small fix
## Issue

The VideoMAE model doesn't work for non-RGB videos (when `num_channels!=3`). I believe this is caused by the harcoded image-net means and stds in the following lines:

https://github.com/huggingface/transformers/blob/d51e7c7e8265d69db506828dce77eb4ef9b72157/src/transformers/models/videomae/modeling_videomae.py#L824L826


## Code to Reproduce

```
import torch
from transformers import VideoMAEConfig, VideoMAEForPreTraining

NUM_CHANNELS = 1

config = VideoMAEConfig(num_channels=NUM_CHANNELS)
model = VideoMAEForPreTraining(config)

pixel_values = torch.rand(1, 16, NUM_CHANNELS, 224, 224)

num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
seq_length = (model.config.num_frames // model.config.tubelet_size) * num_patches_per_frame
bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()

outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
loss = outputs.loss
```
This produces the following error: ```File ".../lib/python3.10/site-packages/torch/functional.py", line 74, in broadcast_tensors
    return _VF.broadcast_tensors(tensors)  # type: ignore[attr-defined]
RuntimeError: The size of tensor a (512) must match the size of tensor b (1536) at non-singleton dimension 2```

Setting NUM_CHANNELS = 3 works fine.

## Potential Fix

Since we don't have a `_DEFAULT_MEAN/STD` for non-RGB images, we can just replace `frames=pixel_values` and disallow `norm_pix_loss=False` if `num_channels!=3`. With this modification, the code works on my machine. I am willing to fix and submit a PR.
Hi,

This is related to #19913. Would be great to open a PR! <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.