I tried to run LSTM-bidir-memory-consumption.py 33 (attached by @tom) on our server. The memory consumption is quite stable.

DummyModule (
(rnn1): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
(rnn2): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
(rnn3): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
(rnn4): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
)
at epoch 0: consuming extra 0.0 MB
at epoch 1: consuming extra 0.0 MB
at epoch 2: consuming extra 0.0 MB
at epoch 3: consuming extra 1.0 MB
at epoch 4: consuming extra 1.0 MB
at epoch 5: consuming extra 1.0 MB
at epoch 6: consuming extra 1.0 MB
at epoch 7: consuming extra 1.0 MB
at epoch 8: consuming extra 1.0 MB
at epoch 9: consuming extra 1.0 MB
at epoch 10: consuming extra 1.0 MB
at epoch 11: consuming extra 1.0 MB
at epoch 12: consuming extra 1.0 MB
at epoch 13: consuming extra 1.0 MB
at epoch 14: consuming extra 1.0 MB
at epoch 15: consuming extra 1.0 MB
at epoch 16: consuming extra 1.0 MB
at epoch 17: consuming extra 1.0 MB
at epoch 18: consuming extra 1.0 MB
at epoch 19: consuming extra 1.0 MB
DummyModule (
(rnn1): LSTM(400, 400, bias=False, batch_first=True)
(rnn2): LSTM(400, 400, bias=False, batch_first=True)
(rnn3): LSTM(400, 400, bias=False, batch_first=True)
(rnn4): LSTM(400, 400, bias=False, batch_first=True)
)
at epoch 0: consuming extra 0.0 MB
at epoch 1: consuming extra 0.0 MB
at epoch 2: consuming extra 0.0 MB
at epoch 3: consuming extra 0.0 MB
at epoch 4: consuming extra 1.0 MB
at epoch 5: consuming extra 1.0 MB
at epoch 6: consuming extra 1.0 MB
at epoch 7: consuming extra 1.0 MB
at epoch 8: consuming extra 1.0 MB
at epoch 9: consuming extra 1.0 MB
at epoch 10: consuming extra 1.0 MB
at epoch 11: consuming extra 1.0 MB
at epoch 12: consuming extra 1.0 MB
at epoch 13: consuming extra 1.0 MB
at epoch 14: consuming extra 1.0 MB
at epoch 15: consuming extra 1.0 MB
at epoch 16: consuming extra 1.0 MB
at epoch 17: consuming extra 1.0 MB
at epoch 18: consuming extra 1.0 MB
at epoch 19: consuming extra 1.0 MB