Visually Inspect your dataset augmentation with the related labels and check if you can overfit with a fine-tuning step:

  
      

      keras.io
  

  
    

Keras documentation: Transfer learning & fine-tuning 3