Any vision model in the library expects pixel_values as input, which should be of shape (batch_size, num_channels, height, width).
ViT (and other models like DeiT, BEiT) expect the height and width to be divisible by the patch_size of the configuration (as these models split up the input image into a sequence of non-overlapping patches, typically of size 16x16 or 32x32).
You can use ViTFeatureExtractor to resize + normalize an image for such a model. Alternatively, you can use torchvision. You can check out the code example of VisionEncoderDecoderModel here.