The bare ViLT Model not working with CUDA ?

@NielsRogge  - Tried in the stock Colab notebook with the latest version of Transformers. Probably the ```to(device)``` is not handled uniformly for Text Embeddings or Image Embeddings?

What I am trying to do: I need to extract the raw hidden_state or pooler_ouput for large batches of image + text pairs. Use it as a feature for training my custom model.

```python
from transformers import ViltProcessor, ViltModel
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
vilt_model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm")
vilt_processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")
vilt_model.to(device)  # Added this ? As it is slow in CPU

from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "hello world"
inputs = vilt_processor(image, text, return_tensors="pt")
outputs = vilt_model(**inputs)

```

Throws

```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper__index_select)
```

Hey! You're getting this error because your model is on GPU while your inputs are not. Could you try casting your inputs to GPU?

```py
inputs = {k:v.to('cuda') for k,v in inputs.items()}
```<|||||>Thanks, I was assuming ViltProcessor will do it for me. Closing