I need to use IITCDIP dataset on a server where I get 6 hours time slot. The data_loader is taking more than 6 hours for caching. So, I can do my training. Is there any way:


load custom dataset with caching (Stream) using script similar to here 2.


Resume the caching process


Cache dataset on one system and use on other system.


Note that I have tried up to 64 num_proc  but did not get any speed up in caching processing.