jmaronas:

How will this estimator of the mean approximates the true mean?


It’s not the true mean. But calculating true mean is infeasible for larger networks and datasets.



 jmaronas:

It is nothing more that a low pass filter of the mean


I guess you can say this with the filter being geometric, but it doesn’t really attribute to understanding BN. In the end, it is just a running average as it is expensive to compute the population average. You can compute the population average, and people have tried that. From experiments, the consensus seem that be that there is no significant gain.



 jmaronas:

I could also use the mean of my validation set.


Using anything from val to train is breaking the purpose of val.