You can check if your loss function implementations are correct by checking if all expected parameters get a valid .grad value after the loss.backward() call. Initially the param.grad attribute is set to None and should contain a gradient tensor after the backward operation.


I’m afraid that your training routine might “learn” to achieve the best loss by setting all weights to zero or negative values. In the end, SGD is trying to lower the loss.