Fail when using FeatureExtractionPipeline for the inference of t5-small
### System Info

- transformers version: '4.24.0'
- platform: Ubuntu 20.04.5 LTS
- Python version: 3.8.10 
- Huggingface_hub version: '0.10.1'
- torch version: '1.13.0+cu117'

### Who can help?

@patrickvonplaten
@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am comparing the embeddings of several very popular HF models (gpt2, bert-base-cased, etc). 
Right now, I want to use the FeatureExtractionPipeline to extract the embeddings of the T5 (small) model. 

```
from transformers import pipeline
feature_extraction = pipeline('feature-extraction', model="t5-small")
sentence = "This is John"
embeddings = feature_extraction(sentence)
```

Error message: 
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/pipelines/feature_extraction.py", line 92, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1074, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1081, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/pipelines/base.py", line 990, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/pipelines/feature_extraction.py", line 70, in _forward
    model_outputs = self.model(**model_inputs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1435, in forward
    decoder_outputs = self.decoder(
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 937, in forward
    raise ValueError(f"You have to specify either {err_msg_prefix}input_ids or {err_msg_prefix}inputs_embeds")
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```
This error occurs at well when using the model at `google/flan-t5-base`. However, when trying the model at `google/mt5-small`, an error arises when loading the pipeline (before trying to us it):
```
>>> feature_extraction = pipeline('feature-extraction', model="google/mt5-small")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 553/553 [00:00<00:00, 393kB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████| 1.20G/1.20G [01:09<00:00, 17.3MB/s]
Some weights of the model checkpoint at google/mt5-small were not used when initializing MT5Model: ['lm_head.weight']
- This IS expected if you are initializing MT5Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing MT5Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 82.0/82.0 [00:00<00:00, 44.2kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 4.31M/4.31M [00:05<00:00, 721kB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 99.0/99.0 [00:00<00:00, 4.37kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/pipelines/__init__.py", line 801, in pipeline
    tokenizer = AutoTokenizer.from_pretrained(
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 619, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    return cls._from_pretrained(
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1932, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 134, in __init__
    super().__init__(
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/venv-tests/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 120, in __init__
    raise ValueError(
ValueError: Couldn't instantiate the backend tokenizer from one of: 
(1) a `tokenizers` library serialization file, 
(2) a slow tokenizer instance to convert or 
(3) an equivalent slow tokenizer class to instantiate and convert. 
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.
```

Is this pipeline not yet prepared for T5 models?


### Expected behavior

I would expect the typical output of the pipeline (a list with a list of embeddings).
Do you have `sentencepiece` installed ?

`pip install sentencepiece` should fix it (as suggests the last line of the error).<|||||>Thanks for the reply.
Installing `sentencepiece` could solve the error when loading `google/mt5-small`.
However, the error when using the other T5 models (`t5-small` and `google/flan-t5-base`) is not related to `sentencepiece`. <|||||>```
(1) a `tokenizers` library serialization file, 
(2) a slow tokenizer instance to convert or 
(3) an equivalent slow tokenizer class to instantiate and convert. 
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.
```

Either you don't have `tokenizers` installed, or `sentencepiece`. It works perfectly in my environement.

Can you try this:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") # This will use `tokenizers` but there is a warning about byte-fallback
# OR 
tokenizer = AutoTokenizer.from_pretrained("google/mt5-small", use_fast=False)  # This will use sentencepiece
```<|||||>Thank you for the response.

I have checked and both `tokenizers` and `sentencepiece` are now installed in my environment.
```
$ python -c "import sentencepiece"
$ python -c "import tokenizers"
```


Additionally, all these tests work:
1. Initializing the tokenizers
```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") # This will use `tokenizers` but there is a warning about byte-fallback
# OR 
tokenizer = AutoTokenizer.from_pretrained("google/mt5-small", use_fast=False)  # This will use sentencepiece
```

2. Executing the fast and slow tokenizers :
```
sentence = "This is John"
tokenizer(sentence)
{'input_ids': [1494, 339, 4040, 1], 'attention_mask': [1, 1, 1, 1]}
```

3. Initializing the feature extraction pipeline with models `google/mt5-small`, `t5-small` and `google/flan-t5-base`. 

However, **the error arises when executing the feature extraction pipeline** with the models `google/mt5-small`, `t5-small` and `google/flan-t5-base`:
```
from transformers import pipeline
feature_extraction = pipeline('feature-extraction', model="google/mt5-small")
sentence = "This is John."
feature_extraction(sentence) # This line breaks

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/pipelines/feature_extraction.py", line 92, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1074, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1081, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/pipelines/base.py", line 990, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/pipelines/feature_extraction.py", line 70, in _forward
    model_outputs = self.model(**model_inputs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1435, in forward
    decoder_outputs = self.decoder(
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/antonio/Documents/Work/LHF/side-projects/compliance/compliance-analyzer/venv/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 937, in forward
    raise ValueError(f"You have to specify either {err_msg_prefix}input_ids or {err_msg_prefix}inputs_embeds")
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

PS: In case it is relevant, my protobuf version is 3.19.1 (I checked it via `$ protoc --version` and via `$ python -m pip show protobuf`)<|||||>Hello, yes `feature-extraction` doesn't really mean much for a encoder-decoder model.

You could use just the encoder, which is probably the closest you want. Keep in mind that models that were not intended for feature extraction will not necessarily be really good at it.

```
pipe.model = pipe.model.encoder # This might depend on a model per model basis, but I think this is what you are looking for, you want just the encoder part of the model.
```<|||||>Thank you for the hack! Closing this issue :)