Adam uses internal running estimates and thus uses more memory than e.g. SGD.
If your GPU is almost full and you call step on your Adam optimizer, these running estimates will be created and might thus yield an out of memory error.