With my limited insight in Pytorchs autograd functionality, I was in the belief, that a computational graph was dynamically created during the forward pass, and hence consumed a lot of memory, while this same memory was gradually freed during the backward pass, where the actual gradients are calculated.
However, when I monitor the memory usage during a forward and backward pass, I see that the forward pass consumes almost no memory, and that the backward pass repeatedly consumes and frees memory in a cyclical manner. I run on a CPU.
Could someone please explain to me why I experience this behaviour?