Hi. For the last year and half I have been using a faster and lower cost normalisation that works perfectly with single sample SGD i.e. minibatchsize=1. This allows much larger deep nets to be trained. Please see this paper https://arxiv.org/abs/1706.03907 61 , Deep Control a simple automatic gain control for memory efficient and high performance training of deep convolutional neural networks.
note from moderators: read this post with skepticism. trusted users have reported that this paper is the same as a known and obvious technique, but upsells it with unknown intent