Hello everyone,
over which dimension do we calculate the mean and std? Is it over the hidden dimensions of the NN Layer, or over all the samples in the batch for every hidden dimension separately?
In the paper it says we normalize over the batch.
In torch.nn.BatchNorm1d hower the input argument is “num_features”. Why would we calculate the mean and std over the different features instead of the different samples?
Thanks in advance