Solved by Kushaj in post #2 
              
              
                The code looks alright, but rather than rationally saying that it is ok, you should do gradient checking to see everything is working as expected. 
PyTorch provides torch.autograd.gradcheck for this purpose docs. Gradient checking is a very good method of debugging mistakes in your model.