Setting tokenizer's return_offsets_mapping to True runs ~10x slower than when set to False
### System Info

- `transformers` version: 4.19.0
- Platform: macOS-12.6-x86_64-i386-64bit
- Python version: 3.9.7
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@SaulLu

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
from datasets import Dataset, load_dataset
import time

# checkpoint = 'bert-base-uncased' # same issue despite different checkpoints
checkpoint = 'roberta-base'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

data = load_dataset("rotten_tomatoes", split='train')

def tokenize_fn(ex):
    return tokenizer(ex['text'])

def tokenize_fn_w_offsets(ex):
    return tokenizer(ex['text'], 
                    return_offsets_mapping=True
                )

def tokenize_fn_w_other(ex):
    return tokenizer(ex['text'],
                    return_attention_mask=True,
                    return_special_tokens_mask=True,
                    return_token_type_ids=True,
                    return_offsets_mapping=False)
                    
start = time.time()
tokenized_data = data.map(tokenize_fn, batched=True, load_from_cache_file=False)
end = time.time()
wall = end - start
print(f'Wall time simple: {wall:.4f} seconds') # took ~0.22 seconds

start = time.time()
tokenized_data_w_offsets = data.map(tokenize_fn_w_other, batched=True, load_from_cache_file=False)
end = time.time()
wall = end - start
print(f'Wall time with other items returned: {wall:.4f} seconds') # took ~0.28 seconds

start = time.time()
tokenized_data_w_offsets = data.map(tokenize_fn_w_offsets, batched=True, load_from_cache_file=False)
end = time.time()
wall = end - start
print(f'Wall time with offsets: {wall:.4f} seconds') # took ~2.5 seconds
```



### Expected behavior

I wouldn't expect the wall time to increase by so much when setting return_offsets_mapping to True. 
cc @Narsil as well<|||||>Hi @tristinb !

Thank you for reporting this issue. I just tested your code snippet on a [google colab ](https://colab.research.google.com/drive/1OV1ICDidCPKFQ9O6XygTzKQKDCMEK8-m?usp=sharing) (with the latest available versions of the libraries) and I observe much smaller time differences, respectively the code takes: 0.9231 seconds, 0.8868 seconds and 1.1825 seconds.

Eventually, it might be worth trying to rerun your code with the latest versions of the libraries (`transformers==4.22.2 datasets==2.5.1 tokenizers==0.12.1`) to see if this reduces the time difference for you too.<|||||>Hi @SaulLu!

Thanks for your response. It looks like bumping `datasets==2.5.1` from 2.4.0 fixed the issue even with an older version of the transformers library. Thanks again!<|||||>So glad this solves your problem! Let me close this issue