For training I have 400 original files times 4 rotations times 2 (flop) which would give about 3200 pictures times random crops (256x256->224x224 - around 1000?). Since this is random not all of them will be used. For validation I have a separate set of 12 pictures - not much, but I think a good start.
Yes all the rotations are included on purpose.
The training and validation datasets are completely independent.
Yes the numbers look like this is indeed the second class. The problem was that output was very, very similar for all validation images - it didn’t seem to depend on input much.
I train the whole network. I am not sure that training just the last layer would be enough as normally the resnet used for image classification should be rotation invariant - quite opposite of what  want to do here.
Good point about trying to pump training data through validation pipeline. Thanks a lot.

While trying to simplify the code for validation I figured out that doing the scaling externally using ImageMagick and removing the scaling code from my program fixes the issue - my validation accuracy is in excess of 80%. Now I need to figure out what was wrong with image resizing using skimage.transform.resize…