Why ViltProcessor couldn't resize image to 384*384?
### System Info


- `transformers` version: 4.27.1
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.8
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import numpy as np
from transformers import ViltProcessor

# create two image
shape_list = [(283, 400), (550, 469)]
fake_imgs = [np.random.randint(0, 255, (3,) + shape) for shape in shape_list]

# preprocess
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")

inputs = processor(
    fake_imgs, 
    ["hello word"] * len(shape_list),
    return_tensors="pt"
    )
# check shape
print(inputs.pixel_values.shape)
# torch.Size([2, 3, 448, 512])

```


### Expected behavior

It seems that `ViltProcessor` is supposed to resize image to 384 * 384 according to `preprocessor_config` (https://huggingface.co/dandelin/vilt-b32-mlm/blob/main/preprocessor_config.json). 
However, the script below gets 448 * 512.
Hi @Yam0214, 

This is because of the padding step that occurs when processing the image - cf [this line](https://github.com/huggingface/transformers/blob/b29fd6971d9cd6ba2a824628effe243f543b8f61/src/transformers/models/vilt/image_processing_vilt.py#L475). This pads all the images in the batch to the largest height and width dimensions in the batch. 

The largest height and width dimensions in the batch are `(448, 512)`. This is because the first image is resized to `(3, 384, 512)` and the second to `(3, 448, 384)`. The reason these images are not resized to `(3, 384, 384)` is because this models preprocessing follows torchvision [resizing logic](https://pytorch.org/vision/main/generated/torchvision.transforms.Resize.html) where: 
* `size` is 384
* `max_size` is `int(1333 / 800 * 384)`<|||||>@amyeroberts Thank you for getting back to me, it's very helpful.