Have you tried to add allow_unused=True in torch.autograd? If you enable it the gradients would be None for the unused variables and then you can look at which variable’s gradients were None and then set that variable’s requires_grad=False or you can set the gradient of all variables with None gradient equal to 0, as you do not need to learn these things.