@lhoestq
Hi lhoestq! Thanks for explaninig how to handle very large dataset.
I have another question about save_to_disk and load_from_disk.
My dataset has a lot of files (#files : 10000) and its size is bigger than 5T.
The workflow involves preprocessing and saving its result using save_to_disk per file (or it takes a long time to make tables).
So it results 10000 arrow files. (total size : 8T)
After this, I edit state.json of one of directories that saved so that its  "_data_files" points arrow files that I made.
The problem is that load_from_disk takes a lot of time (like 6 hours or 3 hours for reloading)
Could I reduce the time it takes? or Could you share another great way to preprocess files?