[Question] How to move and reuse preprocessed dataset?
Hi, I am training a gpt-2 from scratch using run_clm.py.

I want to move and reuse the preprocessed dataset (It take 2 hour to preprocess),

I tried to :
1. copy `path_to_cache_dir/datasets` to  `new_cache_dir/datasets` 
2. set `export HF_DATASETS_CACHE="new_cache_dir/"`

but the program still re-preprocess the whole dataset without loading cache.

I also tried to `torch.save(lm_datasets, fw)`, but the saved file is only 14M.

What is the proper way to do this?
cc @lhoestq the preprocessed dataset should be cached, right?<|||||>Hi ! Yes it should, as long as you didn't change any parameter passed to the `map` function. They must be exactly the same.

Could you open an issue on the `datasets` repo at https://github.com/huggingface/datasets if you want to discuss this caching issue in more details ?<|||||>> Hi ! Yes it should, as long as you didn't change any parameter passed to the `map` function. They must be exactly the same.
> 
> Could you open an issue on the `datasets` repo at https://github.com/huggingface/datasets if you want to discuss this caching issue in more details ?

Sure thanks, the new issue is at here:
https://github.com/huggingface/datasets/issues/2345<|||||>> Hi ! Yes it should, as long as you didn't change any parameter passed to the `map` function. They must be exactly the same.
> 
> Could you open an issue on the `datasets` repo at https://github.com/huggingface/datasets if you want to discuss this caching issue in more details ?

I tried to re-run the example [script ](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py) after a success running (preprocess finished and start training) it still re-preprocess all data.

**Details:**
(1) It re-preprocess data even after showing: `05/11/2021 11:47:01 - WARNING - datasets.builder -   Reusing dataset text (/home/cache/text/default-7083a0557f2cff9e/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5)`
(2) I didn't --overwrite_cache

So, how to reuse preprocessed data? Is there any option I need to open for the scripts? @lhoestq 
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.