Hey @wydwww



 wydwww:

If I set requires_grad=False during training after the DDP ctor, will these parameters be updated anymore, if they still conduct communication?


DDP only builds the communication buffer once (it’s actually twice: 1. using the reverse order of model.parameters, 2. using the autograd order of the first iteration). So DDP will still communicate all parameter’s gradients, even if some of them are marked as requires_grad=False after DDP ctor.


Is there a way to dynamically freeze more parameters and skip their communication during DDP training after constructing the DDP model?


If this is not very frequent (say, once per epoch), you can destroy the DDP instance and create a new one.