Solved by mrshenli in post #2 
              
              
                This is probably due to this PR: https://github.com/pytorch/pytorch/pull/33907 
In v1.5, parameters on replicated models are no longer considered as leaves, as they shouldn’t be. If you really need to access those replicated parameters, you probably can get them from _former_parameters and manually …