The snippet I posted above will zero the gradients of your weight. .backward() will still accumulate gradients in your specific_weight Variable; we’re zeroing out the gradients after a call to .backward() so the optimizer doesn’t use them.