Your dataset is very small, causing your model overfitting. You should try the following options:

Augment your data.
Reduce learning rate or “scheduling lr”.
Study this paper: A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay https://arxiv.org/abs/1803.09820 21.
Take a look at model prediction and compare the groundtruth.
Change metrics to F1 score.

Hope above answers help!
Supachan