Use the target tokenizer in text generation pipeline
# What does this PR do?
A subset of https://github.com/huggingface/transformers/pull/15946
This PR addresses the issue with pipelines in decoding text of different target side tokenizer

This is split from the main PR because this affects the huggingface model hub and API, while the other code is applicable to run in a local copy
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16049). All of your documentation changes will be reflected on that endpoint.<|||||>Hi @AmitMY ,

Thanks for this PR ! It would be nice to get it working with Marian correctly.

Unfortunately, this approach cannot work inside within pipelines.

`.as_target_tokenizer()` does not exist in most tokenizers, and therefore cannot be used (pipeline code is very generic), It's also mutating internal state which is likely going to cause issues in threaded/multiprocessing context.

Here I can think of a better approach if we do indeed need two very distinct tokenizers (it seems to be the case, but the solution for #15946 might change the direction).

The way the pipeline is set, we would need to define two tokenizer, a `preprocess_tokenizer` and a `postprocess_tokenizer` (or `source`/`target`). Each would be unique, and be used accordingly.

By default since most models use the same tokenizer, we could just set both variables to the same object and that should solve the issue at hand.

Before jumping to coding this, I think we should fix the first issue at hand #16050.

Btw, in that issue you mention Mbart, but in the code you modify Marian.


And a note for readers: there's also an open issue to support fast Marian tokenizers https://github.com/huggingface/transformers/issues/15982 which could end up being linked to this work.
<|||||>Not reviewing but commenting for information.

> `.as_target_tokenizer() does not exist in most tokenizers, and therefore cannot be used `

@Narsil `as_target_tokenizer` defined in `PreTrainedTokenizerBase` so it should be available  for all tokenizers. By default it's a no op. cf 
https://github.com/huggingface/transformers/blob/e66743e6c9601a4b12ffc2335a9f60a41d1ca60c/src/transformers/tokenization_utils_base.py#L3405-L3411

> By default since most models use the same tokenizer, we could just set both variables to the same object and that should solve the issue at hand.

Right now I think there are only two models that support two vocabs: FSMT and Marian (#15831), but for these two tokenizers it's not necessary to call `as_target_tokenizer` since they use the target tokenizer in `.decode/batch_decode`<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.