Overlapped offset mapping when manually adding special tokens
### System Info

- `transformers` version: 4.31.0.dev0 (commit 21946a8cf4a273f35ac2f3a53edafc398699f527)
- Platform: macOS-13.2.1-x86_64-i386-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I need to manually insert special tokens in a string so I'm using `add_special_tokens=False` and I encounter decoding inconsistencies between tokenizers that are not present with `add_special_tokens=True`. Here are the test cases with the `openlm-research/open_llama_7b` tokenizer.

```python
>>> tok = AutoTokenizer.from_pretrained("openlm-research/open_llama_7b", legacy=False)
>>> encodings = tok("<s> SYSTEM", add_special_tokens=False, return_offsets_mapping=True)
>>> encodings["input_ids"]
[1, 31822, 18469, 29767]
>>> encodings["offset_mapping"]
[(0, 3), (3, 4), (3, 6), (6, 10)]
>>> tok.decode(encodings["input_ids"])
'<s>  SYSTEM'
```

In the offset mapping, the 2nd and 3rd token are overlapping which is unexpected, and the decoded sequence does not give back the original string, but adds an additional whitespace after the BOS token.

When letting the tokenizer handle the special tokens by itself (`add_special_tokens=True`), the issue is not present.

```python
>>> tok = AutoTokenizer.from_pretrained("openlm-research/open_llama_7b", legacy=False)
>>> encodings = tok("SYSTEM", add_special_tokens=True, return_offsets_mapping=True)
>>> encodings["input_ids"]
[1, 18469, 29767]
>>> encodings["offset_mapping"]
[(0, 0), (0, 2), (2, 6)]
>>> tok.decode(encodings["input_ids"])
'<s> SYSTEM'
```

### Expected behavior

Here is the test case with the LLaMA tokenizer, which works as expected, even when manually handling special tokens.

```python
>>> tok = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy=False)
>>> encodings = tok("<s> SYSTEM", add_special_tokens=False, return_offsets_mapping=True)
>>> encodings["input_ids"]
[1, 28962, 1254, 12665]
>>> encodings["offset_mapping"]
[(0, 3), (3, 6), (6, 8), (8, 10)]
>>> tok.decode(encodings["input_ids"])
'<s> SYSTEM'
```

The offsets are non-overlapping, and the decoding gives back the original string without additional whitespaces.
Hey! 
The `legacy=False` option is pretty much useless for fast tokenizers, so just a FYI here! (the warning is triggered because a slow instance is initialised!) 
The `31822` corresponds to `SPIECE_UNDERLINE = "▁"`. 
The tokenizer on the hub seems to have a few issues already:
```python 
>>> tok = AutoTokenizer.from_pretrained("openlm-research/open_llama_7b", legacy=False, use_fast = False)
>>> tok.encode("                                                ")
[1]
>>> tok = AutoTokenizer.from_pretrained("openlm-research/open_llama_7b", legacy=False, use_fast = True)
>>> tok.encode("                                                ")
[1, 31822, ..........., 31822]
```
Then, there are no `tokenizer.json` files there, which suggest they are using the default Llama converted. But this might not be intended. As you can see, the `huggyllama/llama-7b` is working as expected. I suggest you open an issue on [the original repo](https://huggingface.co/openlm-research/open_llama_7b/discussions)! 
I am not familiar with this model but might have been wrongly converted