Thanks for your advice! I followed it and seemed to have found the source of the problem, which was that I was trying to train VGG from scratch, which apparently is very hard to do. As for the small toy CNN, it was because of the batch normalization. When I removed the batch normalization, the network was able to converge, although I’m not exactly sure why batch normalization affected it that much.