Difference in tokenizer output depending on where `add_prefix_space` is set. 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help

@LysandreJik

## Information

I am using `roberta-base` tokenizer. The tokenization output changes depending on whether `add_prefix_space` is passed into the `from_pretrained` factory as keyword argument or set using property after constructing the .

## To reproduce

Steps to reproduce the behavior:

``` python
from transformers import RobertaTokenizerFast
tokenizer_1 = RobertaTokenizerFast.from_pretrained('roberta-base', add_prefix_space=True)
tokenizer_2 = RobertaTokenizerFast.from_pretrained('roberta-base')
tokenizer_2.add_prefix_space = True

pre_tokenized_inputs = ["Is", "this", "tokenization", "correct"]
tokenizer_1(pre_tokenized_inputs, is_split_into_words=True) 
# {'input_ids': [0, 1534, 42, 19233, 1938, 4577, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}
tokenizer_2(pre_tokenized_inputs, is_split_into_words=True)
# {'input_ids': [0, 6209, 9226, 46657, 1938, 36064, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}
```

## Expected behavior

The addition of prefix space is not working for `tokenizer_2`. Either setting the property should add prefix space to each tokens before splitting into sub-words, or we shouldn't allow it to be set to `True` (raise a exception) after object creation. 
@sai-prasanna This is because RobertaTokenizer is subclassed from GPT2Tokenizer (methods from this class are shared for tokenization) and the params are passed to the parent class only when you instantiate RobertaTokenizer. So, when you change the value of add_prefix_space to True after instantiating RobertaTokenizerFast the value is not propagated to GPT2Tokenizer. 

Link to the relevant code:- https://github.com/huggingface/transformers/blob/eb3479e7cf2c1969897722981bd8e55103f2857f/src/transformers/models/roberta/tokenization_roberta.py#L159<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@frankhart2018 Thanks. Is this the intended behavior or can we try to fix it or warn using some exception or document it atleast.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.