Text Generation Pipeline not using Target Tokenizer
## Environment info

- `transformers` version: 4.18.0.dev0
- Platform: Linux-3.10.0-1160.53.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.1
- JaxLib version: 0.3.0

### Who can help
@patrickvonplaten @patil-suraj @Narsil

## Information
I would like to fine tune MBART, using two different tokenizers: one for the source, and one for the target.

That is mostly implemented, but has a few issues, when the tokenizers are really different.

Specifically, I get UNKs all over the target data, a wrong vocab length, and in the translation pipeline, get rubbish on the output (tokens from the source based on token id from the target)

## Expected behavior

Multiple SPMs should be properly implemented

I have made a specific PR for the pipeline: https://github.com/huggingface/transformers/pull/16049
And a PR for all of it: https://github.com/huggingface/transformers/pull/15946

And they work locally, however the tests don't pass and I'll need help to get this to work.

Not stale - there's a PR
https://github.com/huggingface/transformers/pull/16049<|||||>Gently pinging @Narsil here :-)<|||||>Thanks for the ping.

I think we also need @patil-suraj input on this too.

The tests do not pass right now. Because `tgt_lang` is not defined by default on the tokenizer `AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")`. 

We can probably hack this in pipelines to avoid requiring modifying the `tokenizer`.
```diff
diff --git a/src/transformers/pipelines/text2text_generation.py b/src/transformers/pipelines/text2text_generation.py
index 54a3cfd60..4d3545e4f 100644
--- a/src/transformers/pipelines/text2text_generation.py
+++ b/src/transformers/pipelines/text2text_generation.py
@@ -301,6 +301,11 @@ class TranslationPipeline(Text2TextGenerationPipeline):
                 # translation, XX, to YY
                 preprocess_params["src_lang"] = items[1]
                 preprocess_params["tgt_lang"] = items[3]
+        # Necessary for MBart
+        if "src_lang" in preprocess_params:
+            self.tokenizer.src_lang = preprocess_params["src_lang"]
+        if "tgt_lang" in preprocess_params:
+            self.tokenizer.tgt_lang = preprocess_params["tgt_lang"]
         return preprocess_params, forward_params, postprocess_params

     def __call__(self, *args, **kwargs):
```


@patil-suraj  mentionned that `.decode` should already use the `target_tokenizer` (https://github.com/huggingface/transformers/pull/16049#issuecomment-1064375571) so do we really need to handle it in the pipelines ?

Another thing is that more recent Marian seems to use 2 entirely different tokenizers and this is going to make it harder to respect `added_tokens` and stuff like this IIUC: https://github.com/huggingface/transformers/issues/15982
 
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.