Ah, thanks. I was going to post a movie sentiment net, but that has the drawback of having the embedding layer.
So here is what I did:
I took this example, added the following helper:
# helper function to get rss size, see stat(5) under statm. This is in pages (4k on my linux)
def memory_usage():
    return int(open('/proc/self/statm').read().split()[1])

then I changed the number of epochs to 100 sprinkled gc.collect() and printed the memory_usage() after each epoch.
For me (on python3.5 from Debian with pytorch 0.1.10+821656d from Friday) it added about 1 megabyte of memory per epoch (from ~1.7g RSS to ~1.8g) with the 100 epochs.
I saw similar effects with the sentiment analysis one, but I’m not sure I’ll keep saying that unidirectional LSTM always work…
I pasted the code at https://gist.github.com/t-vi/ea01a032faa1beefc78531e6e292add5 14
@apaszke is that approximately what you had in mind as a test case?