mrshenli:

Note that, although this can reduce the footprint on GPU memory, DDP would still need to communicate the same amount of parameters, as that is determined at DDP construction time. And as those parameters are on CPU, you won’t be able to use NCCL which might cause considerable slow down.


Where do I obtain details corresponding to this particular information? Isn’t only .grad meant to be communicated and the workers applying the updates individually? If my parameters of  theta[t] has only gradients for the particular task, would this help the case? I’m reading the Forward Pass section of Internal Design 1, with find_unused_parameters, it is possible to operate on a subgraph, correct(?). I already have this enabled.