Gradients don’t accumulate in non-leaf nodes. You should register a backward hook on the model: http://pytorch.org/docs/0.3.1/nn.html?highlight=backward%20hook#torch.nn.Module.register_backward_hook 453 to view the gradient of the loss wrt y_pred