It’s a known issue with nn.LSTM module (and nn.GRU too). It’s not a leak, but they use too much memory when using to iterate over inputs. You should use the LSTMCell. It works with CUDA too and should be reasonably fast.