Yes - It seems that your model is good but is learning the wrong features.
I would try data augmentation on your images - if you’re not already doing that.
It took me a long time to appreciate the power of augmentation.
I don’t know what kind of augmentation will work for you - that requires many experiments.
Luckily, PyTorch has a nice set of functions for that.
I hope it helps!