I am using a loss designed by hand so there is no .step() they are just operations written in pytorch. Not sure how that changes things.
According to my pseudocode it seems we had a similar solution? My worry was that I was not sure what mdl.zero_grad() did exactly (besides putting W.grad to zero). Does it also make sure that Op(mdl) does not get included in the computation graph?