Hi @tom @apaszke
I am training a dialog system with opennmt-py https://github.com/OpenNMT/OpenNMT-py 6
My dataset contains 26,265,224 sequence pairs (available here https://github.com/jiweil/Neural-Dialogue-Generation 7)
I observed similar problems: GPU memory consumption remains constant (2.9G), but CPU RAM is almost eaten up.
Initially the cpu memory consumption is around 27G (my dataset is large and it’s acceptable), but after 38.5 hours (around 3.4 epochs) of training, it becomes 53G.
Our main difference is that I am using a unidirectional LSTM. Since I am training models on GPU rather than CPU, but with increased memory consumption on CPU. There might be something to do with the dynamic computational graph, which I believe is created on CPU. It is likely that after each batch of training, the dynamic graph is not destroyed properly. I am not sure if it is because the maximum sequence length are different from each batch and pytorch has to create a new graph per batch.
Oh, here is my torch verison
>>> import torch
>>> torch.__version__
'0.1.10+ac9245a'