I’m not an expert when it comes to the nuances of different optimizers, but from the past I’ve seen that while SGD could yield a lower final loss, making it train/converge might be harder than using a more sophisticated optimizer. As a side note: when trying to debug some code, my default optimizer is always Adam, as it’s usually easy to show if the code has a bug somewhere (e.g. by accidentally detaching tensors) or if the optimization itself is not working due to a bad hyperparameter set.
With that being said, you might want to adapt the learning rate in SGD and see, if this could help.