Have a look at this tutorial 469, where a manual update is described.
If It’s possible to assign your custom gradients to the .grad attribute of all parameters, you could still use an optimizer directly.