The code looks alright, but rather than rationally saying that it is ok, you should do gradient checking to see everything is working as expected.
PyTorch provides torch.autograd.gradcheck for this purpose docs 91. Gradient checking is a very good method of debugging mistakes in your model.