Solved by ptrblck in post #2 
              
              
                Have a look at this tutorial, where a manual update is described. 
If It’s possible to assign your custom gradients to the .grad attribute of all parameters, you could still use an optimizer directly.