You won’t necessarily run out of memory, because if you overwrite them at every forward, you’ll be keeping at most one additional copy of the graph, that’s likely to have its buffers freed if only you called backward on it.
About the original question, you might also want to take a look at register_forward_hook Module method.