Partially fix space only input without special tokens added int the output 
Original issue #3091.

It fixes the issue for all non BPE-based tokenizers. For BPE ones, the output is different from Python and Rust: 

GPT2: 
- Python : `[]`
- Rust: `['Ġ']`

Roberta:
- Python: `[]`
- Rust: `['<s>', 'Ġ', '</s>']`

Rust seems the right one here. I should have a look at Roberta why it include the special_tokens even if not asked to do so.

cc @n1t0 cc @LysandreJik fyi.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>