Actually you’re right. My dataset size is not much. I suspected learning rate to be high, therefore started training with learning rate 1e-3. In that case the model doesn’t fully overfit (training accuracy doesn’t reach around 100%), meaning I didn’t observe such dramatic increases. However, validation accuracy also doesn’t increase accordingly. This is also overfit, I guess, because the model can convey what it seems to learn from training set to validation set.
What made me surprised is that after jumping around global minima with high learning rate, one epoch would be enough to reach almost 100%. It didn’t make sense to me. Thanks for your response.