[HELP NEEDED] Use target tokenizer for all functionality in target side
# What does this PR do?

When using the MarianTokenizer with a different source and target SPMs, it is expected to get the length of the target vocabulary when performing:
```python
with tokenizer.as_target_tokenizer():
    print(len(tokenizer))
```
Along with the expectation to be able to get the target vocabulary, and in translation time to use the target tokenizer.


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Did you write any new necessary tests?

## HELP NEEDED

Tests do not pass, as the token IDs returned change.
However, the IDs are taken directly from the SPM model and not the VOCAB file, and I do not understand why my new values are less correct compared to the ones in the test files.
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15946). All of your documentation changes will be reflected on that endpoint.<|||||>Hi @AmitMY ,

I am not 100% sure not knowing all the internals here, but it's pretty frequent that researchers add code on top of SPM and modify the ids. A behavior which is mimicked in `transformers` so that the actual model works.

So the difference of ids could be just that, and we really the code to stay the same.

I don't have a lot of background on this code to give you a possible better approach, sorry :(<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.