Let me try to explain myself a bit further. I want to perform training (backward and forward pass) using different batch sizes for every layer. I hence need to change the implementation of backward().
Moreover, I have come to understand that it is common to reduce (average out) the losses at the last layer for all samples in a mini-batch and then back-propagate the single value. I would hence like to know if the back-propagation at intermediate layers also takes place in the same manner? Also, do we (if yes, can I see the code) store the intermediate activations of all samples at all layers, or are they reduced as well?