Hi,
Yes one issue is that you are trying to differentiate through the optimizer step. But pytorch optimizers are not differentiable.
You can take a look at libraries like higher 7 on how to go around this.