Add regression tests for slow sentencepiece tokenizers. 
This PR adds regression tests for slow sentencepiece tokenizers. These tests are needed for a refactoring in PR #11716 

## Strange findings
- s2t: `_convert_token_to_id` of `"<s>"` does not give 0

## ToDo
- <s>add test for `_convert_token_to_id`</s> - done, see `test_convert_token_and_id`
- <s>add test for `_convert_id_to_token`</s> - done, see `test_convert_token_and_id`
- <s>add test for `get_vocab`</s> - done
- <s>add test for `vocab_size`</s> - done
- <s>add test for `convert_tokens_to_string`</s> - done, see `test_sentencepiece_tokenize_and_convert_tokens_to_string` in `TokenizerTesterMixin` 
- <s>add test for pickle</s> - is tested in `test_pickle_subword_regularization_tokenizer`
- <s>manual review</s> - done
- <s>fix / add reformer integration test</s> - see https://github.com/huggingface/transformers/pull/11737#issuecomment-850769064 - done
- <s>add typing</s> - done
- <s>add docstrings</s> - done
rebased on master<|||||>This PR is ready for review please. @LysandreJik @sgugger 

The failing test is connected to #11731<|||||>> Cool, thanks a lot for working on these tests! I think that these are already somewhat covered by the common tests, but they're fast and should help identify issues faster.
> 
> However, in order to make sure PR #11716 can be merged, I was mentioning integration tests, rather than regression/unit tests. For example the ALBERT integration test:
> 
> https://github.com/huggingface/transformers/blob/b8344a274fe13b390fa60c74b76117f5ea8144cb/tests/test_tokenization_albert.py#L108-L152
> 
> Those are particularly important when doing refactors that may affect the encoding/decoding aspect of tokenizers.
> 
> I think this is a bit of a larger work though, so we can post "Good first issues" for the SPM-based tokenizers in a first step so that the community may help.

Ok @LysandreJik .
So I will extend the PR and add integration tests for the `_tokenizer` function like the one you linked above to all sentencepiece tokenizers.

Do you think the already written tests can stay as they are? What other steps are needed?<|||||>Hi @PhilipMay, thanks for offering to do it! Feel free to let us know if you would like us to offer some of these to the community, as it can be a bit of work to get every tokenizer tested.

Other than the integration tests, I don't think anything is needed.

Also, you might be interested in rebasing on the `master` branch - we've solved the issue regarding the `run_tests_torch` timing out yesterday so by rebasing you would have reliable CI feedback.<|||||>> Hi @PhilipMay, thanks for offering to do it! Feel free to let us know if you would like us to offer some of these to the community, as it can be a bit of work to get every tokenizer tested.

I was thinking to add integration tests for the tokeinzers that I want to refactor (the sentencepiece). And not foll all tokenizers.

**What about this:** In this PR I add integration tests for the **sentencepiece** tokeinzers only - a full list see here #11417 

After that has been merged I (or you) will open an issue asking for similar tests for all tokenizers.

@LysandreJik what do you think?
<|||||>Yes, sentencepiece tokenizers only, definitely! But even so, that's quite a large number of tokenizers :)<|||||>> Also, you might be interested in rebasing on the master branch - we've solved the issue regarding the run_tests_torch timing out yesterday so by rebasing you would have reliable CI feedback.

Rebased on master - CI is green again. :-)<|||||>@LysandreJik I refactored the tokenizer integration test of albert;

https://github.com/German-NLP-Group/transformers/blob/1894bcc5d116d0107150f4659551e6e21111d736/tests/test_tokenization_albert.py#L127

By adding a util class to `TokenizerTesterMixin`
https://github.com/German-NLP-Group/transformers/blob/1894bcc5d116d0107150f4659551e6e21111d736/tests/test_tokenization_common.py#L186

And also added an integration test to Barthez:
https://github.com/German-NLP-Group/transformers/blob/1894bcc5d116d0107150f4659551e6e21111d736/tests/test_tokenization_barthez.py#L99

What do you think about this "pattern"? Should I continue in that direction and add the other tokenizers?<|||||>I like it, I find it very clean!<|||||>@LysandreJik the reformer tokenizer integration test somehow fails:

```text
Expected :Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between Jax, PyTorch and TensorFlow.
Actual   :Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert provides general-purpose architectures (BERT, GPT-, RoBERTa, LM, DistilBert, LNet... for Natural Language nderstanding (NL and Natural Language Generation (NLG with over  pretrained models in  languages and deep interoperability between ax, PyTorch and TensorFlow.
```

Characters like ")" are missing from the vocab. They are converted to `0` or `<unk>`.
@LysandreJik I just pass in an simpler test text to make the test succeed.
Or should we investigate this stange error to discover a possible hidden bug?<|||||>@LysandreJik while we are here - can we remove this or is it some kind of "open todo"?

https://github.com/huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/tests/test_tokenization_common.py#L178-L184<|||||>@LysandreJik @sgugger as discussed above the suggested integration tests are added to the sentencepiece tokenizers.

CI is green, IMO this is done and ready for merge.

Please have a look at the strange behavior of the reformer tokenizer: https://github.com/huggingface/transformers/pull/11737#issuecomment-850769064

And this question: https://github.com/huggingface/transformers/pull/11737#issuecomment-850776366
<|||||>Pinging @patrickvonplaten regarding the Reformer test.

Regarding https://github.com/huggingface/transformers/pull/11737#issuecomment-850776366 we can just remove this<|||||>> Regarding #11737 (comment) we can just remove this

Done.<|||||>Ok - so this should be ready to be merged so I can continue with #11716 ?<|||||>Thanks again for all your work!