It’s probably not the most efficient way, but you could clone all .grad attributes, zero them out in the model, perform your custom operations, copy them back into the model parameters, and call optimizer.step() when you are done.