Regression pipeline device
# What does this PR do?

Fixes the regression introduced by #21479. Basically doing
```py
from transformers import pipeline

classifier = pipeline("text-classification", device=-1)
```
now fails on v4.27.0 whereas it used to work on 4.26.1

This PR fixes this and adds a test to avoid future regression.

Fixes #22189
Tests pipelines pass for both PyTorch and TF so merging!<|||||>The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22190). All of your documentation changes will be reflected on that endpoint.