`PretrainedTokenizer.return_special_tokens` returns incorrect mask
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.1
- Platform: ubuntu
- Python version: 3.8
- PyTorch version (GPU?): 1.9
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@LysandreJik

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce

```python
tokenizer = transformers.AutoTokenizer.from_pretrained("bert-base-uncased")

text = "foo 雲 bar"
tokens=tokenizer.tokenize(text)
print("tokens              : ", tokens)

inputs = tokenizer(text, return_special_tokens_mask=True)
print("mask                : ", inputs["special_tokens_mask"])
print("mask from input ids : ", tokenizer.get_special_tokens_mask(inputs["input_ids"], already_has_special_tokens=True))
```

Output:

```
tokens                     :  ['foo', '[UNK]', 'bar']
mask                       :  [1, 0, 0, 0, 1]  # [UNK] is ignored!
mask from input ids :  [1, 0, 1, 0, 1]
```

## Expected behavior

`[UNK]` is special token.
`get_special_tokens_mask` is consistent with `__call__`.

Indeed, we have an error in the way the special tokens mask is computed here. See here for the slow tokenizer: https://github.com/huggingface/transformers/blob/3f44a66cb617c72efeef0c0b4201cbe2945d8edf/src/transformers/models/bert/tokenization_bert.py#L297-L299

This seems to also be the case for the fast tokenizer. Would you like to propose a fix? Pinging @SaulLu as it might be of interest to her.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>unstale
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@SaulLu do you have time to look at this?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>**TL,DR :**
To come back on this issue, I would tend to think that in its current state this method (`get_special_tokens_mask`) and this argument (`return_special_tokens_mask` in `__call__`) is very useful. 

Indeed, this behavior is common to all tokenizers (I checked all tokenizers listed in `AutoTokenizer`, I can share a code if you want to have a look) and from my point of view it allows identifying the special tokens that are added by the `add_special_tokens` argument in the `__call__` method (the unknown token is not included in them, see the details section below). 

Nevertheless, I imagine that it is not something obvious at all and that we should perhaps see how it could be better explained in the documentation. Futhermore, we can think about creating a new method that would generate a mask that would also include the unknow token if needed.

What do you think about it ?

**Details:**
The unknow special token does indeed differ from other special tokens in that it is a special token that is essential to the proper functioning of the tokenization algorithm and is therefore not an "add-on" oroptional like all other special tokens. A "unknow" token will correspond to a part of the initial text. 

By the way, the documentation of `get_special_tokens_mask` is `Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model or encode_plus methods.` and the unknow token is not added by the `prepare_for_model` or `encode_plus` methods but by the heart of the tokenizer : the tokenization algorithm.

@tamuhey , could you share your use case where you need to identify the position of unknown tokens? That would be really useful to us :blush: <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Re-opened this issue as I thought a fix needed to be done - but reading @SaulLu's answer I believe the current behavior is correct.

Please let us know if this is an issue to your workflow and we'll look into solutions.<|||||>Hello @LysandreJik, I also encountered the problem. I will use the example in this [issue](https://github.com/huggingface/transformers/issues/16938).

``` Python
import transformers
print(transformers.__version__)
tokenizer = transformers.AutoTokenizer.from_pretrained('roberta-base')

special_tokens_dict = {"additional_special_tokens": ["<test1>", "<test2>"]}
tokenizer.add_special_tokens(special_tokens_dict)

processed = tokenizer("this <test1> that <test2> this", return_special_tokens_mask=True)
tokens = tokenizer.convert_ids_to_tokens(processed.input_ids)

for i in range(len(processed.input_ids)):
    print(f"{processed.input_ids[i]}\t{tokens[i]}\t{processed.special_tokens_mask[i]}")
```

``` Python
Returned output:

0       <s>     1
9226    this    0
1437    Ġ       0
50265   <test1> 0
14      Ġthat   0
1437    Ġ       0
50266   <test2> 0
42      Ġthis   0
2       </s>    1


Expected output:


0       <s>     1
9226    this    0
1437    Ġ       0
50265   <test1> 1
14      Ġthat   0
1437    Ġ       0
50266   <test2> 1
42      Ġthis   0
2       </s>    1
```
My goal is to train a RoBERTa model from scratch with two additional special tokens `<test1>` and `<test2>`. 


For masked language modelling, I don't want customized special tokens to be masked during training. I used `tokenizer` and `DataCollatorForLanguageModeling`. I thought `special_tokens_mask` from tokenizer could [disable special token masking](https://github.com/huggingface/transformers/blob/v4.26.0/src/transformers/data/data_collator.py#L767) in `DataCollatorForLanguageModeling`.
``` Python
processed = tokenizer("this <test1> that <test2> this", return_special_tokens_mask=True)
```
But it didn't recognize `<test1>` and `<test2>`. 

The workaround is 
``` Python
processed = tokenizer("this <test1> that <test2> this")
processed['special_tokens_mask'] = tokenizer.get_special_tokens_mask(processed['input_ids'], already_has_special_tokens=True)
```
It works fine for me on one sentence, but it seems `get_special_tokens_mask` cannot encode in batch, unlike the default tokenizer. 

Do you think it makes sense to modify the behaviour of `return_special_tokens_mask` or to create a new method?
