Hi,
I have been trying to figure out why my code crashes after several batches because of cuda memory error. I understand that probably there is some variable(s) that is not freed because I keep it in the graph. The question is how to debug that kind of thing?
I have wrote a simple line profiler that examines amount of GPU memory on each step, and it seems like on each loss.backwards() step there was massive memory allocation and not all of it frees, it that piles up for a couple of absolutely identical iterations leading to crash. It always crashes on backwards(). I understand that pytorch reuses memory and that is why it might seem like it is not freeing memory, but here is seems like something is indeed leaking.
I tried explicitly del’ing variables (e.g. with data) to mark them for reallocation by torch, no help. I also used CUDA_LAUNCH_BLOCKING=1 to force it to execute things “in place”.
[GPU memory trace] 1.1k
Is there a way to get a memory footprint like “all tensors allocated on GPU”?
Changes are that it is not backwards() fault, e.g. something like
a +10Mb - 30 is free
b +10Mb - 20 is
c +20Mb - 0 is free
clean, b leaks and was not marked as free, 30 freed and not releazed
a +10Mb - no allocation (20 is actually free)
b +10Mb - no allocation, (10 is actually free)
c +20Mb - new large allocation, breaks, even though b is leaking
Anyway, how to track these things?