As you can see, your loss is quite “bouncy”. While it is really low in epoch 4, it grows to a large value, and shrinks again.
This is often an indicator of a high learning rate.
Lower your learning rate to 1e-5 and you’ll get ~99% accuracy on both sets.