Hi,
All the intermediary states do get allocated but they should all be freed when you run in no_grad() (unless you keep reference to them).
And when grad mode is enabled, most of these will be kept alive by the autograd to be able to compute the backward indeed. But nothing else no.