Hello, I’m trying to figure out what happen behind the scenes in DistributedDataParallel when it comes to parameters that do not require a gradient. I cannot find a clear answer on this in the documentations.
Assume we have three layers: A --> B --> C. Suppose that A and B both have their requires_grad flag as False. If this model is wrapped in DistributedDataParallel, will there be any process communication that needs to be done in the backward pass of layer A or C? Specifically with the sharing of gradients.
My problem is that I have a large model and I am extremely bottle necked by process communication. I would like to freeze some of my layers such that less gradients need to be shared. I understand that depending on my model, the computation cost may be the same but I really need to bring down the communication cost.