If the numbers you mentioned are observed via nvidia-smi, then it is not an accurate depiction of the actual memory usage since pytorch use a caching allocator http://pytorch.org/docs/0.3.0/notes/cuda.html#memory-management 57. Moreover, the cudnn may choose different algorithms basing on different architectures. Since your model contains conv layers and an LSTM, it uses cudnn heavily.