Question in load datasets  of train seq2seq model 
### System Info


- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-149-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit-gandhi @sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I Was change official example scripts data load to my own code:
`    if data_args.data_path is not None:
        print(data_args.data_path)
        raw_datasets = load_dataset("audiofolder", data_dir=data_args.data_path, cache_dir=model_args.cache_dir)
        raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
        raw_datasets = raw_datasets["train"].train_test_split(test_size=0.005, shuffle=True)`

The processing process is as follows: 
1、Resolving data files
2、Downloading data files
3、Computing checksums
4、Downloading data files
5、Extracting data files
6、Generating train split
What caused the significant difference in step six？
![image](https://github.com/huggingface/transformers/assets/19569322/557c99cd-21fa-4805-a1a7-83fe8ecd8530)


### Expected behavior

load fast，need at least 1000+

> Generating train split: 388773 examples [32:24:45, 1574.04 examples/s]
I think this is just CPU workers spinning up, you can verify by asking on the datasets repo where you can get dedicated datasets help: https://github.com/huggingface/datasets<|||||>Yes，I have already asking on the datasets repo but no response yet<|||||>Feel free to gently ping `mariosasko` (he's usually very responsive)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.