Hello.
I was wondering how accurate is the running average and running std that lot of people (including pytorch batch norm functions does)
i understand that for each batch the running average (r_avg) mean is computed as:
r_avg = r_avg0.1 + 0.9batch_mean
where batch_mean is the actual mean of the batch.
How will this estimator of the mean approximates the true mean?. That would be for all the training data compute the mean estimator:
avg = sum (all_data_values) / (all_data)
Remark that if I stop the training and compute the means for each training batch, mu_1, mu_2 (for batch 1, batch 2) and do the mean of this means that is not the true mean, that is (if N is the number of batches):
(mu_1+mu_2+…+mu_N )/ N \neq avg
I now this because once I had to implement the mean of a huge dataset that could not be kept in memory.
Viewing this estimator from a signal processing perspective:
r_avg = r_avg0.1 + 0.9batch_mean
It is nothing more that a low pass filter of the mean, that is, assuming our mean is noisy this kind of estimator will make a more stable mean (we can view the momentum in SGD like this, and it basically reduce the oscillations in the gradients). This make me think this is the actual reason, however i do not see if it is an appropriate estimator to see if training is going well (when looking at hyperparameters).  I could also use the mean of my validation set.
Thanks.