clean_up_tokenization_spaces=True won't clean up spaces
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-5.10.0-051000-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.10.2+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (the same on CPU)
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik, @Narsil, @SaulLu 


## Information

Model I am using (Bert, XLNet ...): BERT (`bert-large-cased`)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
encoded = tokenizer("This thing costs £4.56")
decoded = tokenizer.decode(encoded["input_ids"], clean_up_tokenization_spaces=True)
print (decoded)
```
Real output: `[CLS] This thing costs £4. 56 [SEP]`

I tried it also with NER pipelines and other text inputs.
Additional example: got `[CLS] ( including once - a - week tapping ) [SEP]` instead of `[CLS] (including once-a-week tapping) [SEP]`


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected output: `[CLS] This thing costs £4.56 [SEP]`.

I expected the tokenizer to cleanup all the spaces introduced. Is there any different way to do so? Am I missing some trivial parameter?
Hi @MorenoLaQuatra,

This is normal, as punctuation is considered as token splitter by this tokenizer. The `clean_up` is only meant to remove duplicate spaces, and normalize other space characters into the regular ascii one.

What are you trying to achieve ?

Using `.offset_mapping` is in general, could help you understand more what's going on without modifying the original string.

```python
from transformers import AutoTokenizer

sentence = "This thing costs £4.56"
tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased")
encoded = tokenizer(sentence, return_offsets_mapping=True)
for _id, (start, stop) in zip(encoded.input_ids, encoded.offset_mapping):
    print(f"Token {_id} ({start}-{stop}) comes from {repr(sentence[start:stop])}")
```
```
Token 101 (0-0) comes from ''
Token 2023 (0-4) comes from 'This'
Token 2518 (5-10) comes from 'thing'
Token 5366 (11-16) comes from 'costs'
Token 1069 (17-18) comes from '£'
Token 2549 (18-19) comes from '4'
Token 1012 (19-20) comes from '.'
Token 5179 (20-22) comes from '56'
Token 102 (0-0) comes from ''
```

This allows you to understand what is happening without having to "recreate" the original string (`decode` cannot in general recover the original string)<|||||>Thank you for the explanation. I'm trying to build a custom NER-like system using token classification. I leverage offset_mapping during training to identify the token class. While trying to integrate it in a NER-like pipeline (TokenClassification) I found out it was generating entities from the tokenized version.

However, from your answer, I think it would be better to use char indexes to identify the correct span in the input sentence right?  <|||||>> However, from your answer, I think it would be better to use char indexes to identify the correct span in the input sentence right?

I cannot answer in general as I don't know if you have access to the original string for instance.
But if you do have access to it, then `offset_mapping` will be closer to what you expect most of the time. `decode` is a best effort way to represent what the model has seen, but it cannot in general output exactly what you sent in.

The reason is that `tokenizer.encode` is destructive and looses information. A simple example is that some tokenizer start by `.lower()` so we cannot in general recovering the capitalization. The same goes for spaces, `decode` will try to add them where they belong, but it cannot work 100% of the time.
Here around punctuation for instance You would like to get `"£4.56"` but you would want `"Hi. How are you doing ?"` (with the extra space after the dot). Since it looks the same to the input ids, `decode` has to make a choice and just use one form.<|||||>Ok, I got exactly your point. Thank you and sorry for the issue, I'll close it.<|||||>No worries, you're not alone having this issue, we're looking into how to communicate overall better the information.<|||||>@Narsil   I also have trouble in the same problem. And your explanation is informative. But is there any way to customize this tokenizer behaviors?