@apaszke Right. Thank you for your patience with me here!
So here is a small thing hitting 100 MB extra memory consumption after 12-18 “epochs” with bidirectional LSTM and not without.

  
      gist.github.com
  
  
    https://gist.github.com/t-vi/60515a24e1cbc4dc87897a6d8c224698 67
LSTM-bidir-memory-consumption.py
#!/usr/bin/python3

import torch
from torch.autograd import Variable
import torch.nn as nn
import gc

# helper function to get rss size, see stat(5) under statm. This is in pages (4k on my linux)
def memory_usage():
        return int(open('/proc/self/statm').read().split()[1])
This file has been truncated. show original
output.txt
DummyModule (
  (rnn1): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
  (rnn2): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
  (rnn3): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
  (rnn4): LSTM(400, 400, bias=False, batch_first=True, bidirectional=True)
)
at epoch 0: consuming extra 0.0 MB
at epoch 1: consuming extra 1.9 MB
at epoch 2: consuming extra 22.8 MB
at epoch 3: consuming extra 23.1 MB
This file has been truncated. show original




  
  
    
    
  
  


I don’t think it is the contiguous call + summation as it also happens when I double the input dimension of the 2nd-4th LSTM layer instead.