Gabriel_Barello:

Ahhh, is it that f(x) is the thing it throws out on the forward pass (and has to recompute) and instead of checking is anything in f needs gradients it only checks if x needs a gradient?


Kinda. Layer’s output itself goes to next layer, and may be early released or not depending on what is done there. But output’s “history” references other tensors, sometimes these are the only live references.
Re: checkpoint behaviour. I think it is mostly an implementation quirk, related to how autograd.Function works. But there also may be an assumption that you’d exclude at least the first gradient inducing operation from checkpointing, as you have no “history” (=potentially disposable tensors) at that point, so rerun is just harmful.