Different results between `AlbertTokenizer` and `AlbertTokenizerFast` modules with a new `spiece.model` file
Hello!

I would like to ask your opinion about a tokenizer behavior. In a project, I have to train a new tokenizer to re-pretrain an Albert model. I don't know if I did something wrong (and if I did, I'd love to know!) but for the moment a text is not tokenized in the same way with `AlbertTokenizer` and `AlbertTokenizerFast`.

Thanks a lot for your time in advance :smile: 

## To reproduce

Steps to reproduce the behavior:

1. Training a tokenizer with [sentencepiece library](https://github.com/google/sentencepiece). The resulting tokenizer is saved under the name `spiece.model`. I can share it if needed.
2. Assuming that only the `spiece.model` file is in the root. Run the following blocs of code: 
```python
tokenizer_dir_path = "."
text = "a\n b"
```
Cell:
```python
albert_tokenizer = AlbertTokenizer.from_pretrained(tokenizer_dir_path)

print("ids", albert_tokenizer.encode(text))
print("ids -> ids_token",albert_tokenizer.convert_ids_to_tokens(albert_tokenizer.encode(text)))
```
Output:
```bash
ids [2, 1842, 5132, 3]
ids -> ids_token ['[CLS]', '▁a', '▁b', '[SEP]']
```
Cell:
```python
albert_tokenizer_fast = AlbertTokenizerFast.from_pretrained(tokenizer_dir_path)

print("ids", albert_tokenizer_fast.encode(text))
print("ids -> ids_token",albert_tokenizer_fast.convert_ids_to_tokens(albert_tokenizer_fast.encode(text)))
```
Output:
```Bash
ids [2, 1127, 266, 3157, 3]
ids -> ids_token ['[CLS]', '▁a', '▁', '▁b', '[SEP]']

```
Cell:
```python
sp = spm.SentencePieceProcessor(model_file=os.path.join(tokenizer_dir_path, "spiece.model"))

print("ids", sp.encode(text))
print("ids -> ids_token", sp.id_to_piece(sp.encode(text)))
```
Output:
```bash
ids [1127, 3157]
ids -> ids_token ['▁a', '▁b']
```
Other variations:
I also tried to instantiate the tokenizer like this `AlbertTokenizerFast(vocab_file=os.path.join(tokenizer_dir_path, "spiece.model"))`.

## Expected behavior

I expected to have the same result with the modules: `AlbertTokenizer` and  `AlbertTokenizerFast`. In particular, I did not expect "\n" to be tokenized by "_" in the case of `AlbertTokenizerFast`.

## Environment info

- `transformers` version: 4.5.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Albert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Hey @SaulLu, thanks a lot for the detailed issue. I managed to reproduce the issue by using another tokenizer on the hub, `codegram/calbert-tiny-uncased`, which has the same issue.

@n1t0 helped me identify the issue, and we have a fix in #11367. Would you mind trying it out and let me know if it fixes your issue? The correct behavior is that of the slow tokenizer, there's an excess space in the fast tokenizer encoding.

You can either checkout the branch and install from source - or you can install the following in your env:
```
pip install -U git+https://github.com/huggingface/transformers@fix-albert-converter
```<|||||>Hey @LysandreJik !

Thank you very much for your detailed answer! I tested the fix provided on #11367. The tokenization is now the same with the previous example `text="a\n b"` which became `['[CLS]', '▁a', '▁b', '[SEP]']` ! :+1: 

Unfortunately, I have a similar inconsistency that was not resolved with the following example: 
```
text="\n"
```

Cell:
```python
albert_tokenizer = AlbertTokenizer.from_pretrained(tokenizer_dir_path)
print("ids -> ids_token",albert_tokenizer.convert_ids_to_tokens(albert_tokenizer.encode(text)))
```
Output:
```bash
ids -> ids_token ['[CLS]', '[SEP]']
```
Cell:
```python
albert_tokenizer_fast = AlbertTokenizerFast.from_pretrained(tokenizer_dir_path)
print("ids -> ids_token",albert_tokenizer_fast.convert_ids_to_tokens(albert_tokenizer_fast.encode(text)))
```
Output:
```Bash
ids -> ids_token ['[CLS]', '▁', '[SEP]']

```
Cell:
```python
sp = spm.SentencePieceProcessor(model_file=os.path.join(tokenizer_dir_path, "spiece.model"))
print("ids -> ids_token", sp.id_to_piece(sp.encode(text)))
```
Output:
```bash
ids -> ids_token []
```<|||||>If it can help, I started to compare the tokenizer.json files between the one contained in `albert-base-v2` and the one obtained by running the command :
```
albert_tokenizer_fast = AlbertTokenizerFast.from_pretrained("SaulLu/albert-bn-dev")
albert_tokenizer_fast.save_pretrained("./albert-bn-dev-fast")
```

An extract of `albert-base-v2` `tokenizer.json`'s file is: 
```
{
    "version": "1.0",
    "truncation": null,
    "padding": null,
    "added_tokens": [
        {
            "id": 0,
            "special": true,
            "content": "<pad>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 1,
            "special": true,
            "content": "<unk>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 2,
            "special": true,
            "content": "[CLS]",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 3,
            "special": true,
            "content": "[SEP]",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 4,
            "special": true,
            "content": "[MASK]",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        }
    ],
    "normalizer": {
        "type": "Sequence",
        "normalizers": [
            { "type": "Replace", "pattern": { "String": "``" }, "content": "\"" },
            { "type": "Replace", "pattern": { "String": "''" }, "content": "\"" },
            { "type": "NFKD" },
            { "type": "StripAccents" },
            { "type": "Lowercase" },
            {
                "type": "Precompiled",
                "precompiled_charsmap": "..."
            }
        ]
    },
    "pre_tokenizer": {
        "type": "Sequence",
        "pretokenizers": [
            { "type": "WhitespaceSplit" },
            { "type": "Metaspace", "replacement": "▁", "str_rep": "▁", "add_prefix_space": true }
        ]
    },
    "post_processor": {
        "type": "TemplateProcessing",
        "single": [
            { "SpecialToken": { "id": "[CLS]", "type_id": 0 } },
            { "Sequence": { "id": "A", "type_id": 0 } },
            { "SpecialToken": { "id": "[SEP]", "type_id": 0 } }
        ],
        "pair": [
            { "SpecialToken": { "id": "[CLS]", "type_id": 0 } },
            { "Sequence": { "id": "A", "type_id": 0 } },
            { "SpecialToken": { "id": "[SEP]", "type_id": 0 } },
            { "Sequence": { "id": "B", "type_id": 1 } },
            { "SpecialToken": { "id": "[SEP]", "type_id": 1 } }
        ],
        "special_tokens": {
            "[SEP]": { "id": "[SEP]", "ids": [3], "tokens": ["[SEP]"] },
            "[CLS]": { "id": "[CLS]", "ids": [2], "tokens": ["[CLS]"] }
        }
    },
    "decoder": {
        "type": "Metaspace",
        "replacement": "▁",
        "str_rep": "▁",
        "add_prefix_space": true
    },
    "model": {
        "unk_id": 1,
        "vocab": [...]
    }
}
```

An extract of `albert-bn-dev-fast` `tokenizer.json`'s file is: 
```
{
    "version": "1.0",
    "truncation": null,
    "padding": null,
    "added_tokens": [
        {
            "id": 0,
            "special": true,
            "content": "<pad>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 1,
            "special": true,
            "content": "<unk>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 2,
            "special": true,
            "content": "[CLS]",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 3,
            "special": true,
            "content": "[SEP]",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false
        },
        {
            "id": 4,
            "special": true,
            "content": "[MASK]",
            "single_word": false,
            "lstrip": true,
            "rstrip": false,
            "normalized": true
        }
    ],
    "normalizer": {
        "type": "Sequence",
        "normalizers": [
            { "type": "Replace", "pattern": { "String": "``" }, "content": "\"" },
            { "type": "Replace", "pattern": { "String": "''" }, "content": "\"" },
            { "type": "NFKD" },
            { "type": "StripAccents" },
            { "type": "Lowercase" },
            {
                "type": "Precompiled",
                "precompiled_charsmap": "..."},
            { "type": "Replace", "pattern": { "Regex": " {2,}" }, "content": " " }
        ]
    },
    "pre_tokenizer":{"type":"Metaspace","replacement":"▁","add_prefix_space":true},
    },
    "post_processor": {
        "type": "TemplateProcessing",
        "single": [
            { "SpecialToken": { "id": "[CLS]", "type_id": 0 } },
            { "Sequence": { "id": "A", "type_id": 0 } },
            { "SpecialToken": { "id": "[SEP]", "type_id": 0 } }
        ],
        "pair": [
            { "SpecialToken": { "id": "[CLS]", "type_id": 0 } },
            { "Sequence": { "id": "A", "type_id": 0 } },
            { "SpecialToken": { "id": "[SEP]", "type_id": 0 } },
            { "Sequence": { "id": "B", "type_id": 1 } },
            { "SpecialToken": { "id": "[SEP]", "type_id": 1 } }
        ],
        "special_tokens": {
            "[CLS]": { "id": "[CLS]", "ids": [2], "tokens": ["[CLS]"] },
            "[SEP]": { "id": "[SEP]", "ids": [3], "tokens": ["[SEP]"] }
        }
    },
    "decoder": { "type": "Metaspace", "replacement": "▁", "add_prefix_space": true },
    "model": {
        "type": "Unigram",
        "unk_id": 1,
        "vocab": [...]
    }
}
```
By replacing the content of `pre_tokenizer` key with `{
        "type": "Sequence",
        "pretokenizers": [
            { "type": "WhitespaceSplit" },
            { "type": "Metaspace", "replacement": "▁", "str_rep": "▁", "add_prefix_space": true }
        ]
    }` in the `albert-bn-dev-fast/tokenizer.json` file, then the Fast tokenizer returns the same results as the slow on the 2 discussed examples. :smiley: 

<|||||>Hi @SaulLu, sorry for getting back so late on this. 

I think you've stumbled upon another difference between the slow and fast tokenizers which would need to be patched within `tokenizers` directly.

Is it of great importance to your task? While it is an issue, I would argue it is quite low priority as such a use-case seems rare and the issue doesn't seem to have a huge impact. Please let me know if you think this should be bumped up. 