Solved by my3bikaht in post #2 
              
              
                Correct mean and variance of the whole dataset needed to do proper normalization, but batchnorm has access only to batch part of data each step. That’s why batchnorm is saving intermediate values and updates them each batch-step. 
Maxpool only returns max value in the pool in input data. Thus, you d…