Hi thanks for quick reply!
I am using pytorch-lightning 1.1.7 on top of the torch 1.7.1.  I don’t directly call torch api’s, but using lightning’s Trainer, and model.fit .
The lightning underline indeed printed out the model parameters, all three are the same,  (they all rounded to thousand though).
The weird thing is that, the training worked very well on first machine, which has two gpus.  The problem happened to the second machine, which has 3 gpus.  But even after i removed 1 gpu from the the machine, training with 2 gpus on this machine still fails w/ replicas error.