[Benchmark] Tokenizers as Collate functions vs normal in loop tokenizing
Apologies if this is more of a bug than it is a benchmark, just wanted to check if it was normal for tokenizer as collate function to run twice as slow. [Here's the colab notebook](https://colab.research.google.com/drive/1dqnM3pWIZgUMTrk42RJrvby6-ZZCHNRO?usp=sharing).
Maybe @sgugger has an idea.<|||||>Please explain to us what the problem is, the Colab notebook you link is not public.<|||||>Sorry @sgugger, should be public now but basically the problem is that using a collate function slows down dataloader:
```python
sentence_dl = DataLoader(
    sentence_ds,
    BATCH_SIZE,
    num_workers=NUM_WORKERS,
    shuffle=False,
    drop_last=False,
    pin_memory=True,
)
# fast
for batch in tqdm(sentence_dl):
    x = collate_fn(batch)
```

```python
sentence_dl = DataLoader(
    sentence_ds,
    BATCH_SIZE,
    num_workers=NUM_WORKERS,
    shuffle=False,
    drop_last=False,
    pin_memory=True,
    collate_fn=collate_fn,
)
# slow
for batch in tqdm(sentence_dl):
    continue
```

Where sentence_ds is just a `Dataset` created from the sentences (strings) in "snli" dataset and collate function is:
```python
tokenizer = AutoTokenizer.from_pretrained(LANGUAGE_MODEL)

class CollateFn:
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer
        
    def __call__(self, x):
        return self.tokenizer(
            x, 
            max_length=MAX_TEXT_LENGTH, 
            truncation=True, 
            padding="max_length", 
            return_tensors="pt"
        )
    
collate_fn = CollateFn(tokenizer)
```<|||||>Understood. Yes, it's very likely that using the multiprocessing on the dataloders side slows down the multiprocessing of the "fast" tokenizer.

FYI, for best speed, I would recommend passing 1,000 samples at once to the tokenizer and preprocessing once and for all beforehand.