I am back with an update:

I am using AlexNet pretrained. It fails to converge at all (CE remains higher than 1.0)
When I unfreeze the convolution layers it starts to learn, but only so slightly.
I am using a small dataset (4000 images), and when training it achieves about 0.1 to 0.01 CE however upon evaluation it is less than 30% accurate (correct out of total evaluated images)
I’ve played with the hyperparameters quite a bit, learning rate, momentum, L2, epocs, etc.
I am using RGB and not grayscale, and the default 224x224 input. I normalise and resize to 224x224

I am wondering:

Why am I getting better results when I try to classify valence instead of expression labels (3 classes instead of 7)
Why am I stuck at similar accuracy regardless of hyperparameter changes (about 25% to 30%)
Should I be increasing the training size?
Is my image preprocessing enough? E.g., resize and normalise?
Does using less output classes make such a big difference?
Why does the CE yo-yo up and down in most cases, is this an indication of failing to converge?

@ptrblck I appreciate any help or advise here, I know my questions are not so much about pytorch.
Regards,
Alex