I was about to do the same kind of question, when I saw this post.
I had (relatively) small model built in Keras (Theano backend) that only took ~200-300 Mb of GPU memory. However, the same model implementation in pytorch takes 2200Mb of GPU memory. I’m using Adam, but then again I was using it in Keras as well…