[`Pix2Struct`] Fix slow test
# What does this PR do?

Fixes: https://github.com/huggingface/transformers/actions/runs/4538560416/jobs/7997667941 

This test was currently failing due to the fact that I forgot to add `.to(torch_device)` on a slow test

cc @sgugger 
