In sub-path training, such as NAS，there are two path A and B need to be optimized. for example, the optimizing order is B->A:
1.when using a single-gpu, everything works well if we set require_grad=False for unselected path，
2.but when using DistributedDataParallel, we need set find_unused_parameters=True in this condition，and because of  grad sync，for example, when optimizes path A in step2, the parameters‘ grad in path B will be 0 rather then None
Here comes the problem: optimizer chooses to update params which grad is not None. If we set momentum in SGD or something else in other  optimizer, when optimizing path A in step 2, because we have already optimized path B once，so parameters in B have momentum_buffer, although path B’s grad is 0,  it will also be updated by the buffer which is incorrect!
Even we set require_grad=False for path B, but once path B is optimized, its grad will be 0 rather then None.

image1712×786 65.7 KB