Batch size is 32 (I have also tried 16) for training and 64 for validation.
The graph in the original post was from the more complex mode which might explain the higher loss. The validation loss in the simpler example show above is not so different (example shown below).
I have now tried without batchnorm layers which resulted in the same issue. The attached image is of that run.
modelRun21797×624 34.7 KB
What is interesting looking at the model training is even from the start the output is always a single class per batch (the same class repeated for every sample). The predicted class will just change slightly during training which would probably explain the one jump in accuracy.