Hi @ptrblck, I’m having an issue along the same lines.
I’m using an Alexnet model from scratch, having more or less followed this breakdown Finetuning Torchvision Models — PyTorch Tutorials 1.2.0 documentation. My code can be seen here AlexNet-Filter-Fun/train_model.py at 3cd29eb55b3b11e2668211a5e4d129dc0dfbe359 · NeilFranks/AlexNet-Filter-Fun · GitHub.
As you recommended, I’m trying on a small data sample (two classes), but I’m getting absolutely no movement (in validation, it guesses the same class every time, resulting in an accuracy of 0.5).
Even in the tutorial I linked, at the bottom you can see they examined training models from scratch, and even THEIR accuracy stayed completely stagnant from the beginning.
What am I missing? I suspect it’s something to do with my optimizer?

Edit: Now I’m sure it’s to do with the optimizer; upon finding someone else trying to implement it the same way as the original Alexnet paper, they too say it “doesn’t train”, and when I use the optimizer they used instead (Adam), it trains okay!
Could you help a beginner like me understand why the Alexnet paper’s optimizer just doesn’t train?