The first time my training scripts are run, the dataset is ‘compiled’ in the Dataset class. For instance, I do work often with medical data and then it is possible I add another dataset, and I want to cut this data into tiles of a specified shape. These files are then cached to the SSD, so next time the compilation phase is skipped.
However, when I use DistributedDataParallel all processes will do this. I could check the rank of the process, and only allow rank == 0 to execute this, but then the other processes will crash because they will find an empty dataset. Is there a way I can tell the second process to wait before it starts training?