I recently encounter a situation where some of the model parameters will not be updated during certain iterations. The unused parameters are those are not in computation graph (after backward(), the gradients of those unused parameters is None)
I find the training result is different when I do not have those unused parameters. The only reason I could think of is the optimizer Adam and maybe other adaptive learning rate optimization. Because those optimizer take all model parameters as input when initialized but I only update part of them.
Anyone knows how should I resolve this issue. I am not sure the reason I propose above is correct or not.