Solved by ptrblck in post #2 
              
              
                You could use hooks to manipulate the gradients for each parameter (its .grad attribute), backward hooks, or if needed you could also create a custom Adam implementation by reusing the PyTorch implementation.