Solved by tom in post #2 
              
              
                Due to the normalization implicit in Adam, the learning rates aren’t comparable to SGD, and you probably use a much too high learning rate for SGD. 
Best regards 
Thomas