Maybe I wasn’t clear. The model classifies the training data more or less perfectly, but the randomly chosen 20% holdout images for validation aren’t very accurate.
There is no problem with balance of samples across the classes.
Also, the training is quite fast (a few seconds per epoch) because I am preloading the training images into the GPGPU.
I am trying some experiments with dropouts now. Definitely slows down the learning. Meanwhile, I welcome other suggestions. Thanks!