[SpeechT5] Fix HiFiGAN tests
# What does this PR do?

SpeechT5HiFiGAN tests added in the PR #21702 failed in the CI daily run: https://github.com/huggingface/transformers/actions/runs/4248850253/jobs/7388500325

This PR fixes the torch devices ✅
_The documentation is not available anymore as the PR was closed or merged._