Hi,
I see that for most of the implementations in pytorch, it is common to calculate the net output of a mini-batch by running it through a model, and then calculating and normalizing the loss. This implies that the loss (on which backward() is called) seems to be an arithmetic mean of all the losses produced by individual samples of the mini-batch at the output layer.
My question is, does this process also take place at every layer? Do we normalize the loss at a given layer, use that same loss for all activations of that layer and compute the gradients for back-propagation? It would be really helpful if someone could provide a reference to the modules in pytorch that actually do this when backward is called.