Our cases may be quite similar. Still, it is probably much more complicated than just a pytorch error.
As @fmassa said it make sense that the backward phase GPU requirement is really tied with the architecture of your model. By this I mean not only the number of parameters.
For example, my model has way less parameters than another one that works with 8GB of RAM while mine can’t. The reason behind it is - I think - that, because of intradecoder attention. i.e. at each step I compute attention of the h_t with all previous h_t’ (with t’<t). This links the gradient of the current step to every previous steps. The same goes for other variables.
I’m not sure if I’m clear, but, by studying my code, the paper, and other papers with respective implementation it make sense that the model i’m trying to implement requires more RAM. Maybe it’s the same for you.
Again, for reference, I’m talking about A Deep Reinforced Model for Abstractive Summarization 6.
If someone is interested by implementing it we could collaborate.