I’ve also been trying to understand the how things are happening at the lower level of the cudnn API calls. It seems that the loss and weight update is responsibility of the optimizer. In the case of cuda, that will just handle the output and gradient computation.