I’ve implemented a custom dataset which generates and then caches the data for reuse.
If I use the DataLoader with num_workers=0 the first epoch is slow, as the data is generated during this time, but later the caching works and the training proceeds fast.
With a higher number of workers, the first epoch runs faster but at each epoch after that the dataset’s cache is empty and so overall there is not much gain.
My understanding is that each worker keep getting initialized with a fresh dataset each time and hence the problem (the exact mechanism is not clear to me as I can see the dataset is only constructed once). Now I don’t expect the cache (or any data) to be shared between processes, but I would expect each worker to keep its cache throughout the training. Instead it seems like new workers are created at each epoch and so the desired state is lost. Is there a way to prevent that from happening?