Anecdotal and orthogonal, but I’ve successfully trained object detection convnets (homebrew that’s somewhere between OverFeat and ResNet Faster R-CNN) with single-image-batches and batchnorm at every layer. Not sure why it works (I would expect it to fail) but it trains just fine.