I wanted to create a model that would detect whether image is rotated 0, 90, 180 or 270 degrees (classify into one of 4 classes).
I took the following approach based on https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 18 :

Took about 400 self made photos, rescaled to minimum dimension of 256 pixels, took random crop of 224x224, random rotation (affects label), added flop (does not affect label). Then the dimensions are swapped, the pixel values are divided by 256 (to be in 0-1 range) and normalized (to be centered around 0).
Used resnet18 as base model and added fc layer with 4 outputs.
I am trying to teach my network something for 25 epochs, which on a CPU is quite time consuming 


Unfortunately while the training accuracy goes slightly above 90% the validation accuracy does not show the model is useful at all. I tried to check how much the model deviates from expected output “by hand” and it seems that the model does not depend much on the input - when looking at values assigned to a class on output they are very similar across validation images (eg. -0.2, 0.6, -0.4, 0.1).
Therefore I have the following questions:

Do you think the approach makes sense for this scenario?
Do you think I have enough training data?
How can I debug my training/validation process? What can go wrong?