Incosistent behaviour between fast and slow RoBERTa tokenizers
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.0.dev0
- Platform: Linux-4.15.0-122-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.3.4 (cpu)
- Jax version: 0.2.19
- JaxLib version: 0.1.70
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
- tokenizers: @LysandreJik
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

I was testing a fix for #9933 and when debugging the test for RoBERTa tokenizers I found that fast and slow return different results for the test and neither of the results are what I expected.

The code snipet below is a copy of `tests.test_tokenization_roberta.RobertaTokenizationTest.test_special_tokens_mask`.

As you can see from the output, slow tokenizer outputs `<unk>` ids eventhough the flag state to not return special tokens.
Also, the special tokens mask returned doesn't take into account the `<unk>` tokens as if they weren't special tokens.

On the other hand, the fast tokenizer doesn't output those tokens since `<unk>` is defined as a special token, as expected.
However, when adding special tokens the `<unk>` tokens are not added at all.

So I am wondering which behaviour is correct since niether seems to be 100%?

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior: run the following code

```python
"""Based on test from test_tokenization_roberta.RobertaTokenizationTest.test_special_tokens_mask """

import os
import json
import tempfile
import shutil

from transformers import RobertaTokenizer, RobertaTokenizerFast
from transformers.models.roberta.tokenization_roberta import VOCAB_FILES_NAMES

# Setup
vocab = [
    "l",
    "o",
    "w",
    "e",
    "r",
    "s",
    "t",
    "i",
    "d",
    "n",
    "\u0120",
    "\u0120l",
    "\u0120n",
    "\u0120lo",
    "\u0120low",
    "er",
    "\u0120lowest",
    "\u0120newer",
    "\u0120wider",
    "<unk>",
]
vocab_tokens = dict(zip(vocab, range(len(vocab))))
merges = ["#version: 0.2", "\u0120 l", "\u0120l o", "\u0120lo w", "e r", ""]
special_tokens_map = {"unk_token": "<unk>"}

tmpdirname = tempfile.mkdtemp()
vocab_file = os.path.join(tmpdirname, VOCAB_FILES_NAMES["vocab_file"])
merges_file = os.path.join(tmpdirname, VOCAB_FILES_NAMES["merges_file"])
with open(vocab_file, "w", encoding="utf-8") as fp:
    fp.write(json.dumps(vocab_tokens) + "\n")
with open(merges_file, "w", encoding="utf-8") as fp:
    fp.write("\n".join(merges))

kwargs = {}
kwargs.update(special_tokens_map)
kwargs.update(do_lower_case=False)
slow_tok = RobertaTokenizer.from_pretrained(tmpdirname, use_fast=False, **kwargs)
fast_tok = RobertaTokenizerFast.from_pretrained(tmpdirname, use_fast=True, **kwargs)

sequence = "Encode this."

print("Slow tokenizer:")
print(f"  Encoding: {slow_tok.encode(sequence, add_special_tokens=False)}")
encoded = slow_tok.encode_plus(sequence, add_special_tokens=True, return_special_tokens_mask=True)
print(f"  Encoding with special: {encoded['input_ids']}")
print(f"  Special tokens mask: {encoded['special_tokens_mask']}")
print("Fast tokenizer")
print(f"  Encoding: {fast_tok.encode(sequence, add_special_tokens=False)}")
encoded = fast_tok.encode_plus(sequence, add_special_tokens=True, return_special_tokens_mask=True)
print(f"  Encoding with special: {encoded['input_ids']}")
print(f"  Special tokens mask: {encoded['special_tokens_mask']}")
shutil.rmtree(tmpdirname)
```
The output I get from running this code:
```
file /tmp/tmpros2tpnz/config.json not found
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
file /tmp/tmpros2tpnz/config.json not found
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
file /tmp/tmpros2tpnz/config.json not found
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Slow tokenizer:
  Encoding: [19, 9, 19, 1, 8, 3, 10, 6, 19, 7, 5, 19]
  Encoding with special: [20, 19, 9, 19, 1, 8, 3, 10, 6, 19, 7, 5, 19, 21]
  Special tokens mask: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
Fast tokenizer
  Encoding: [9, 1, 8, 3, 10, 6, 7, 5]
  Encoding with special: [20, 9, 1, 8, 3, 10, 6, 7, 5, 21]
  Special tokens mask: [1, 0, 0, 0, 0, 0, 0, 0, 0, 1]
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Both tokenizers should output the same results:
```
Slow tokenizer:
  Encoding: [9, 1, 8, 3, 10, 6, 7, 5]
  Encoding with special: [20, 19, 9, 19, 1, 8, 3, 10, 6, 19, 7, 5, 19, 21]
  Special tokens mask: [1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1]
Fast tokenizer
  Encoding: [9, 1, 8, 3, 10, 6, 7, 5]
  Encoding with special: [20, 19, 9, 19, 1, 8, 3, 10, 6, 19, 7, 5, 19, 21]
  Special tokens mask: [1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1]
```
<!-- A clear and concise description of what you would expect to happen. -->

Maybe @SaulLu do you have an idea here? :-)<|||||>Hi @patrickvonplaten @SaulLu, Can I help in resolving this issue?
But as I'm quite new to this so I will need some guidance in where should I start from.<|||||>@ofirzaf, thanks for the detailed issue!

**About the missing id corresponding to the unknown token for the fast tokenizer** 

Yes, I can see why the fast tokenizer does not take into account the unknow token. 

Since the original RoBERTa tokenizer does not need this token (since it is byte-based and contains the exhaustive list in its vocabulary), when converting from the slow to the fast version of the tokenizer the information of the unknown token you added in the kwargs is not passed (precisely, the information is "lost" in [this method](https://github.com/huggingface/transformers/blob/v4.9.2/src/transformers/convert_slow_tokenizer.py#L216)). 

@ofirzaf , in the short term, if you need to initialize the fast tokenizer from the slow version files, you can do so: 
```python
special_tokens_map = {"unk_token": "<unk>"}

kwargs = {}
kwargs.update(special_tokens_map)
kwargs.update(do_lower_case=False)
fast_tok = RobertaTokenizerFast.from_pretrained(tmpdirname, use_fast=True, **kwargs)
fast_tok.backend_tokenizer.model.unk_token = special_tokens_map["unk_token"]
fast_tok.save_pretrained("local_tok")

fast_tok = RobertaTokenizerFast.from_pretrained("local_tok", use_fast=True)
```

@sourabh112, It's very kind of you to offer your help. However, in the immediate future, I'm still not sure we want to change this behaviour without ensuring that there will be no adverse effects (the GPT2 tokenizer being reused in several places) knowing that these are initially tokenizers that should not need the unknown token.

@patrickvonplaten, @LysandreJik and @sgugger  do you have an opinion on this? Could we at least add a short-term warning? 

**About the `return_special_tokens_mask`** 

It seems to me that this behavior is common to all tokenizers. Special tokens are tokens added to transform the tokenized text into a format compatible with the input expected by the model. The unknown token is different in that it is a necessary token for the tokenisation algorithm.
<|||||>@SaulLu Thanks for the reply.

I agree that this issue shouldn't occure when using the tokenizer for the reasons you mentioned.
I think, however, that the test should be fixed to reflect that. As I mentioned, the example I brought here is straight from the library's built in tests.

Regarding the special tokens mask, if the `<unk>` token shouldn't be considered as a special token, shouldn't it be removed from the special tokens list of the tokenizer?

In the OP I mentioned another issue I wanted to fix, can you take a look at the issue and the proposed fix and tell me if this is something you think is worth fixing/contributing or the team doens't think this is an issue?

Thanks<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.