I also have the same problem. I implement a ResNet+LSTM model. It’s funny that it seems adam is more unstable, or converge really slow while SGD only takes 6 epoches to converge.
But most of the report claimed that Adam should converge more quickly.