If your dataset is really quite small you could be in a few shot domain.
In that case you can take look at some tips in:

  
      

      arXiv.org
  

  
    

How to fine-tune deep neural networks in few-shot learning? 11

  Deep learning has been widely used in data-intensive applications. However,
training a deep neural network often requires a large data set. When there is
not enough data available for training, the performance of deep learning models
is even worse...