to revise this slightly…I believe I understand what the “problem” actually may be. Since it is more likely that the processes finish each epoch at different times rather than all simultaneously at a point in time, and each process is calling optimizer.step() asynchronously then saving the parameters at the end of an epoch…it would make sense that the values of the weights would be slightly different due to this.
So at the beginning of the training, all processes should have the same initial weights…but over the course of training it is likely that the processes will not actually have copies of the model that are identical to those of the other processes, but the parameters will generally be more or less similar.