Hi,
I recently started looking into using pytorch for an active learning project and have some queries in using distributed data parallel. As per my understanding, distributed data parallel creates a process per gpu to run the model. Active learning approaches usually have two stages: annotation and training. In the annotation stage, I need to run the model on the unannotated samples and get annotations for selected samples and in the training stage, the model needs to be trained on the annotated dataset.  I am trying to use DistributedDataParallel but am not sure how to.
My workflow will roughly look like:

Run the model on unannotated samples (across all gpus)
rank samples using the output from previous step based on some criterion (main worker, rank 0)
add new samples to the training dataset (all workers’ data loaders need to be updated) and then run training.

This cycle will repeat till a budget is hit (that budget needs to be synced across all processes in order to be able to terminate them).
I am not clear on how I get the output of the model from all processes (step 1) back into the main process, run things like the sample ranking (step 2) on the main process while the other processes wait for the input from the main process before starting the training cycle again. Also, how do I sync variables like budget across processes?  It would be great if I can get a pointer to a resource which helps me understand this more.  I have looked at the imagenet training script in the github repo but that didn’t help me understand this process.
Thanks!