shirui-japina:

Usually we use Mini-batch Gradient Descen-like to update the parameters. Which means we get the average value of the loss function to update parameters.
Is the process of averaging (do reduction (e.g. mean)) equivalent to this process?


Yup   We have a loss for each sample in your batch, then mean/average that to get one loss-float which we can use to backpropagate. (edit: doesn’t need to be mean, but other reduction techniques such a sum)