Hi Tom, thank you for your comment! I also was expecting this.
My VGG has actually the batch norm in his architecture, I think that this would be the issue.
One question: Does BatchNorm gives good result with very small batch size (different from 1), like batch_size=2?