Ah, that makes sense from a memory-saving point of view for inference. But the input is needed for backpropagation to update the weight matrices. So it has to be kept somewhere during training? Presumably it would be in the

module

part of:
def hook_fn(module, grad_input, grad_output)