You could use hooks to manipulate the gradients for each parameter (its .grad attribute), backward hooks, or if needed you could also create a custom Adam implementation by reusing the PyTorch implementation 1.