Hi,
Sorry for opening up this old thread. Can you please explain why initialization is necessary when we are using batch normalization? Batch normalization seems to normalize along the batches and reduces the problem of the “Mean length scale in final layer” as described in How to Start Training:The Effect of Initialization and Architecture 9 by Hanin and Rolnick. Also, by intuition, batch norm should actually reduce the mean to zero and standard deviation to one before the layer output is fed to later layers. As such, this should prevent any kind of slowing down due to zigzagging as described in Efficient Backprop by LeCun