It’s hard to tell what might be causing the failure in training you are seeing, but I would think it depends on the hyperparameters you are using as well as the overall training routine.



 punto:

Could you help a beginner like me understand why the Alexnet paper’s optimizer just doesn’t train?


I would try to check the reference implementation (if there was one), as I’m sure there will be codes which reproduce the original claims of the paper (or come close to them). Alexnet is probably one of the more important models, so it would be surprising if the paper isn’t reproducible (unless the authors have a corrected version and the original paper doesn’t mention it).