Hi,
I’m having RAM related issues with a new model I implemented.
So I just checked how much memory was needed at each steps. I was running the model and monitoring process max RAM usage, when I exit at different steps.
I tried with both SGD and ADAM

Right after .forward(): 11GB
Right after computing loss: same
Right after .backward(): failed: Out Of memory, >32GB (including swap)

Isn’t it weird that the .backward consumes that much RAM? i.e. more than the model itself?
My model itself may not be optimized with respect to memory i.e. maybe do I keep too much informations etc. It contains some non-parameter Variable (i.e. with requires_grad=False). But I don’t see how those thing may impact the backward pass…
Any ideas?