Could you check, if the model parameters are getting a valid gradient by printing the .grad attributes of them after the first backward call?
Also, why do you need to use retain_graph=True here?
loss.backward(retain_graph=True)

As a general guidance I would try to overfit a small data sample first (e.g. just 10 samples) and make sure the model is able to overfit it.