Your dataset is quite small. Are you training your model from scratch? If so, could you try to use a pretrained model and fine tune it?
Since your data is so small you might need a lot of regularization, e.g. weight decay.