Yeah, sometimes Adam don’t work with moderately high learning rate, so in that case I usually use sgd with low learning rate(0.001 or 0.01, these lrs are lower enough for sgd) to avoid nan with high momentum like 0.8 or 0.9 for fast convergence. You can try this as well.