[examples] run_clm re-processes dataset on every run
developing with `run_clm` is difficult since its startup is very slow - it rebuilds the dataset on each start.

@VictorSanh says it started to do that recently...

I think it's because it has to chunk the existing dataset into smaller pieces, it's a slow start everytime and it doesn't save these results.
So the original dataset has already been preprocessed, but it's not good enough for `run_clm.py`.

So I'm thinking perhaps for dev needs we need a dataset with short  <512 entries? and then it could use it w/o additional preprocessing?

But I could be wrong I haven't investigated the reason for the slow start.

to reproduce:
```
USE_TF=0 python  examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name "stas/openwebtext-10k" \
    --output_dir output_dir \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --max_train_samples 1000 \
    --max_eval_samples 200 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --num_train_epochs 1 \
    --warmup_steps 8 \
    --block_size 64 \
    --fp16 \
    --report_to none
```

So look at the tqdm bars before training starts to see the symptom. And this is already a very truncated dataset.

@VictorSanh, @sgugger 

The dataset caching is all relying on the datasets library, so the issue should probably be tracked here. Especially if this is a new change: since there was no change I'm aware of in `run_clm` recently it may be coming from a change there.<|||||>Thank you! I will ask on the `datasets` side.<|||||>you scooped me Sylvain.

I downgraded datasets to `1.5.0` and printed `tokenized_datasets.cache_files` (L335):

> `{'train': [{'filename': '/home/victor/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b/cache-c6aefe81ca4e5152.arrow'}], 'validation': [{'filename': '/home/victor/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b/cache-97cf4c813e6469c6.arrow'}]}`

while the same command with the latest version of datasets (actually starting at `1.6.0`) gives:
> `{'train': [], 'validation': []}`

Does it ring any bell @lhoestq ?<|||||>OK, moved this to `datasets` https://github.com/huggingface/datasets/issues/2387
<|||||>Reopening and bringing it back here:

According to this https://github.com/huggingface/datasets/issues/2387#issuecomment-845781874

we need to change examples to add `keep_in_memory=False` - load_dataset otherwise there is no caching.

here:
https://github.com/huggingface/transformers/blob/223943872e8c9c3fc11db3c6e93da07f5177423f/examples/pytorch/language-modeling/run_clm.py#L233

<|||||>ok, `datasets` reverted the in-memory-datasets by default in master, so this is no longer a problem.