Xformers is not installed correctly.
### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

``` python 
from transformers import pipeline
pipe = pipeline("text-classification", model="roberta-base", device=0)
```

Edit: I know this model isn't trained for the "text-classification" task, I get the same problem with a private model I fine tuned.

Results in the message

```
...
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
```

But I'm using torch==2.0.1 and [memory-efficient-attention](https://huggingface.co/docs/diffusers/optimization/fp16#memory-efficient-attention ) states "If you have PyTorch 2.0 installed, you shouldn’t use xFormers!"

The message is confusing - I have torch 2.0 installed and pipeline is for inference. This message doesn't occur if I use `AutoModelForSequenceClassification.from_pretrained`

### Expected behavior

The documentation or the warning message are inconsistent.
It looks like the `pipeline` is back to importing every model (this message comes from trying to access an unrelated model). I'll have a look later this week. You can ignore that warning in the meantime, it's irrelevant.<|||||>Should be fixed by the PR linked above.