I am currently profiling the memory-consumption of my model. My code is essentially:
log_probs = model.log_prob(inp)
loss = (-1)*log_probs.mean()
loss.backward()

I have ~4 gig of GPU memory (3980MB) before calling backward and ~8 gigs (8882MB) after, but this is suprising to me. Pytorch maintains the whole computational graph before calling backward, which it should be able to free while computing it. I have 2 281 048 parameters, where for each parameter a gradient is computed. I am not sure why I now occupy ~8 gigs of GPU memory?
If I am correct, this means:
(8 gigabytes) / (4 bytes) = 2 000 000 000 floats
which are a lot of floats and more than I would expect.
If this is not expected, is there any way in which I could figure out what’s going on? For example, get the memory consumption per module?
EDIT: The reason for my investigation is that I get an OOM when scaling to a higher-dimensional dataset and I am currently looking into reducing the memory-requirement.