Solved by mrshenli in post #2 
              
              
                Hey @ayalaa2, DistributedDataParallel’s (DDP) ctor would go through all parameters and skip the ones whose requires_grad=False. So, there won’t be communication on those grad, but you will have to set their require_grad field before passing it to DDP. After the ctor, changing the requires_grad attri…