Thanks for confirming that this is the right way to go about it, it’s just not supported yet 
In the meanwhile I’ll just store the gradients away and manually add them back before I call the optimiser. Results are still the same, so that’s probably a tuning issue.