Fix device issue in `SwiftFormerModelIntegrationTest::test_inference_image_classification_head`
# What does this PR do?

The title says everything 😄 
_The documentation is not available anymore as the PR was closed or merged._