Hey @Xiaopeng_Li, which version of PyTorch are you using? After https://github.com/pytorch/pytorch/pull/33907, the replicate method is no longer supposed to be used this way. It will only replicate non-leaf parameters, and as a result, replicated_model.parameters() will be empty. Can you double check if this is the case in your dev env?
If you really need this, you can try to access the _former_parameters attribute in replicated models. See the code below. But there is no guarantee on how long this attribute can stay.

  
      github.com
  
  
    pytorch/pytorch/blob/0edbe6b063d2525ceaf89f0c603f6e35b3118686/torch/nn/parallel/distributed.py#L347-L357

def parameters(m, recurse=True):
    def model_parameters(m):
        ps = m._former_parameters.values() \
            if hasattr(m, "_former_parameters") \
            else m.parameters(recurse=False)
        for p in ps:
            yield p

    for m in m.modules() if recurse else [m]:
        for p in model_parameters(m):
            yield p