Thanks for an answer.
Yes, you are right, I was actually bypassing the DDP model by manually calling “.module” of DDP wrapped model. I think that’s why model parameters across GPUs were different.
Now I created explict forward function in MyModel class, and use it instead of “func_a, func_b”, model parameters across GPUs were same. I think that DDP wrapped model trigger synchronization with forward pass