Sorry for bumping this old thread, but I couldnt find any more information about this!
Is it necessarily bad that your BatchNorm statistics are computed per mini-batch?
How would this influence your training, and is there a way to prevent/change this behaviour?