The specific optimizer shouldn’t impact the ability to converge vs not in most cases. It should be possible to use any of the common optimizers for this task, it’s usually a matter of getting all the details right and searching over your hparams (if you aren’t starting from known good defaults).
That said, AlexNet is difficult to train and most adaptive optimizers tend to be more forgiving in challenging situations or non-optimal hparams. So, first q would be why Alexnet? Using a net that has normalization layers (ie BatchNorm) and residual connections will make it significantly easier to train. And if it must be Alexnet, you probably need to dig through the original impl and make sure your weight init is close to the original (which is likely not the case anymore with the pytorch default impl), you also need to use the correct batch size vs LR. I think the original was 128 batch size for .01, but not clear if that 128 factored in the 2x GPU so it miight be 256 and .01 equivalent. Being off with SGD even a small amount can mean instability with a net like this.