I’m facing the same problem, actually its worse in my case.
forward consumes 1.5G of memory and backward uses 11G of memory. I’m using a LSTM encoder and decoder. 11x increase doesn’t seem normal
Edit - The length of encoder/decoder are fairly substantial (140 for encoder, 40 for decoder). Could this be due to computing intermediate gradients on each step during backward, while for forward it doesn’t copy the weights like a persistent RNN?
Edit 2- I’ve run a few tests. Decreasing decoder input length reduces memory consumption. So, its likely that backward memory balloons because of gradient computation at each step. But the main problem I’m having is for each backward call extra memory is allocated. So after a few training steps I get an OOM error.
I’ve made sure to delete the graph after each step by deleting the loss I backprop on. Not sure where the leak is from really