Thanks for the reference! Besides matching Python, initializing the weights to all-ones intuitively seems like the right thing to do – although, technically, this is not suggested in the original paper, and I haven’t seen any study of the effects of BatchNorm initialization on convergence/performance. I’m still puzzled by the huge negative impact this had on my results!