Hi, all
I’m trying to use the distributed package for multi-gpu training. Because of the way the code is written, the master process does all the initialisations (creating model replicas, optimisers etc.). From pytorch source code, it seems like during forward pass, all model replicas will be synced with the one running in subprocess with rank 0. Does that mean I could just initialise one optimiser for subprocess 0 and only update the parameters of the first model replica?
Thanks,