I’ve plotted the activation mean and variance for the very first layer over 1000 batches. The output of this layer is passed to the first BatchNorm layer so these are the means and variances the BatchNorm layer is using. Is this distribution problematic? It seems they follow a sensible distribution.