Thanks for responding so fast. I have tried the code you suggested and I am stuck without any improvement. My testing and evaluation data shows the same accuracy. The accuracy doesn’t drop after a while. If it does it’s only by a few points and then it comes back to previous levels. (46 - 48%) I am able to overfit on the training data as you said. I have also tried lowering the learning rate after a while by stopping the program and starting it again with new learning rate. This helps the training but still not the eval or test.
Are you suggesting that I should add more dropout now? Is the goal to bring the training down or the eval up? I would of course prefer the latter.