Hi,
The issue is that you want to backprop through the optmizer step?
Well the issue is that the optimizers in pytorch are not differentiable  So I would recommend using a library like higher 1 that is built do differentiate through gradient updates!