Please correct me if my understanding is wrong.
Your current solution is to have a second script that monitors the training script. If the second script notices that the training script is beginning to be bottlenecked by dataloading, it will jump into action and begins pre-processing data to be utilized. Is this correct?
If this is the case, then yes PyTorch does offer some functionality to make this easier. The argument I would point you towards is num_workers.
You can increase num_workers to spawn more processes to fetch the data (which involves pre-processing the data). If you’re finding that you cannot keep up with the training loop, then you can increase the number of workers. Assuming your pre-processing script is able to keep up, then you should achievable a similar affect with increasing the number of workers.
Please let me know if there’s something I misunderstood about this problem