As your model cannot overfit this small data set, you should try to debug it and play around with hyperparameters etc. to make sure it’s able to do so.
E.g. you could remove some layers and work with a very simple model first (even a single conv layer could be a valid experiment).