Thanks @ptrblck
Playing with momentum did help get this training correctly, although after sweeping through a large range of momentums, the best model on the validation set is without using batch norm at all.
Thanks again, consider this solved