The problem is in AI there is something called overfitting. Basically this happens when your model is able to memorize the train dataset but does poorly on the validation or test datasets. Because your model has learned the exact training set it cannot generalize. Here 4 is a good article on it. To fix it you could look at some more transformations found here 4. I would recommend random rotation, color jitter, and random resized crop but you can mess around with them all to see which do best.