I’m not familiar with the mentioned loss function, but if you already checked the gradients, it would at least mean that the computation graph is not detached accidentally.
You could check, if the relu activation is needed at the model output, as this seems to be unusual (of course it might still fit your use case).