Solved by Ce_Wang in post #5 
              
              
                Thanks for your advice! I followed it and seemed to have found the source of the problem, which was that I was trying to train VGG from scratch, which apparently is very hard to do. As for the small toy CNN, it was because of the batch normalization. When I removed the batch normalization, the netwo…