Possible bug in `prepare_for_model` when using fast tokenizers
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.1
- `tokenizers` version: 0.9.3
- Platform: Linux
- Python version: 3.7.2
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

Also confirmed with `transformers==4.2.2` & `tokenizers==0.9.4`

### Who can help
tokenizers: @n1t0, @LysandreJik
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

I am building my own data preprocessing script which requires me to first know the number of tokens in each sentence before, then match sentences pairs and prepare them as input for a model, in this case BERT. I would like to use the fast tokenizer to speed things up in large datasets, however, I encounter the next assertion problem which is not true since I do provide `return_special_tokens_mask=True` in the function call.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

The following code snippet reproduces the problem for me:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)
s1 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("Halt! Who goes there?"))
s2 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("It is I, Arthur, son of Uther Pendragon, from the castle of Camelot. King of the Britons, defeator of the Saxons, sovereign of all England!"))
tokenizer.prepare_for_model(s1, s2, return_special_tokens_mask=True)
```

I get the following assertion error when running the code:
```
AssertionError                            Traceback (most recent call last)
<ipython-input-5-889164fb3ae8> in <module>
      2 s1 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("Halt! Who goes there?"))
      3 s2 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("It is I, Arthur, son of Uther Pendragon, from the castle of Camelot. King of the Britons, defeator of the Saxons, sovereign of all England!"))
----> 4 tokenizer.prepare_for_model(s1, s2, return_special_tokens_mask=True)

~/p/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in prepare_for_model(self, ids, pair_ids, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis, **kwargs)
   2724         if return_special_tokens_mask:
   2725             if add_special_tokens:
-> 2726                 encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
   2727             else:
   2728                 encoded_inputs["special_tokens_mask"] = [0] * len(sequence)

~/p/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in get_special_tokens_mask(self, token_ids_0, token_ids_1, already_has_special_tokens)
   3031         """
   3032         assert already_has_special_tokens and token_ids_1 is None, (
-> 3033             "You cannot use ``already_has_special_tokens=False`` with this tokenizer. "
   3034             "Please use a slow (full python) tokenizer to activate this argument."
   3035             "Or set `return_special_token_mask=True` when calling the encoding method "

AssertionError: You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument.Or set `return_special_token_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer. 
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
For the fast tokenizer to provide the same output as the slow (python) tokenizer:
```python
{'input_ids': [101, 9190, 999, 2040, 3632, 2045, 1029, 102, 2009, 2003, 1045, 1010, 4300, 1010, 2365, 1997, 21183, 5886, 7279, 7265, 7446, 1010, 2013, 1996, 3317, 1997, 19130, 4140, 1012, 2332, 1997, 1996, 28101, 5644, 1010, 4154, 2953, 1997, 1996, 28267, 1010, 11074, 1997, 2035, 2563, 999, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'special_tokens_mask': [1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

<!-- A clear and concise description of what you would expect to happen. -->

## Possible Fix
Make the following change in:
https://github.com/huggingface/transformers/blob/d1b14c9b548de34b6606946482946008622967db/src/transformers/tokenization_utils_base.py#L2855
Since we know beforehand if the special tokens were added and the text pair was already concatenated with or without special tokens, I think the following change will be valid, however, I didn't test it beyond my own use case
```python
        if return_special_tokens_mask:
            if add_special_tokens:
                encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(sequence, already_has_special_tokens=True)
            else:
                encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
```
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.<|||||>I ran into the same problem (I am also working on building inputs for pretrained models). @n1t0, @LysandreJik Could you give a comment on this?

I think the problem arises from the fact that `BertTokenizerFast.get_special_tokens_mask` calls `PreTrainedTokenizerBase.get_special_tokens_mask` whereas `BertTokenizer` overrides `get_special_tokens_mask` method. It seems that problem will be solved if the fast tokenizer also overrides the method. Am I missing something?<|||||>Might also be a solution, havn't looked into it. I think that the fix I suggested in the original post might resolve problems like this in other tokenizers aswell and not only in BertTokenizer.

I wonder if fixing this will be a welcomed contribution to the library or the wontfix tag is there for a reason?

@LysandreJik