It’s hard to tell what might be causing the failure in training you are seeing, but I would think it depends on the hyperparameters you are using as well as the overall training routine.

Can you elaborate on what might be the culprit in the training routine or other hyperparameters, then, if simply changing the optimizer from torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=0.0005) to torch.optim.Adam(params=model.parameters(), lr=0.0001) seems to alleviate my problems (where there was zero improvement in accuracy, there is now much improvement!) in the small, two-class sample run?
Additionally, I found the more or less “canon” implementation. I see they mention using SGD in the wiki
--mini 128	train using SGD with minibatch of 128 examples

but aside from that I cannot untangle where/how it’s being done. I see the learning rate stuff in the code, but nothing related to SGD with “momentum of 0.9” as specified in the paper and mentioned in passing in the wiki.
Here’s where I think they do all the training cuda-convnet2/gpumodel.py at 3238bf0367f63eb370e897b9e5714794cb67ddc2 · akrizhevsky/cuda-convnet2 · GitHub ; perhaps the SGD stuff is obfuscated behind the C++ model they loaded up?? It’s hard for me to decipher what this code is doing, tbh
    def import_model(self):
        lib_name = "cudaconvnet._ConvNet"
        print "========================="
        print "Importing %s C++ module" % lib_name
        self.libmodel = __import__(lib_name,fromlist=['_ConvNet'])

Anyways, from what else I’ve seen in looking at how people have implemented Alexnet in PyTorch, they consistently seem to use Adam. For instance, this piece of code which I did not see/haven’t looked at until just now, after I ran into this problem in my own implementation alexnet-pytorch/model.py at d0c1b1c52296ffcbecfbf5b17e1d1685b4ca6744 · dansuh17/alexnet-pytorch · GitHub.
I think I/we are missing something nuanced about the optimizer, I’m just not sure what… any help would be appreciated (for now, though, I’ll use Adam  )