AndrewSoul:

I think the gradients of ModelOutputs(net, layer=40) is different from net.features[40].weight.grad.


That might be the case and I suspect you might be registering the weight parameter and overriding it with the bias parameter later, since both will match the condition.
Could you check the shape and see, if you are in fact storing the bias gradient?