@mrshenli Thanks for the reply. I’m using 1.4.0. In this version the replicated_model.parameters() do exist, and I understand that in 1.5.0 it no longer exist.
As I mentioned, it works for several iteration of training, but crashes at some point. Every time it crashes at different iterations, pretty weired. Am I doing reduce_grads  correctly?