Hi ! Currently the CSV loader doesn’t leverage multithreading nor multiprocessing.
This is something we are working on, see issue [load_dataset] shard and parallelize the process · Issue #2650 · huggingface/datasets · GitHub 6 which should allow to parallelize the conversion over multiple csv files.
However I’m not very familiar tools that allow to do multithreading on single files though. So if you have any idea/direction that could speed up the conversion of csv files to arrow, feel free to share it here