The most detailed account is in @ptrblck and my imaginary book which focuses on PyTorch details like that (and tries to avoid explaining deep learning).
The other day I wrote an extension that shows the autograd graph in Jupyter at the end of each cell (or when you want) to create the visuals and facilitate interactive exploration. But didn’t think to include .grad and I’ll admit that I don’t think it’ll say “memory leak”, so it would be an imperfect solution for the thing here.
If the present book is any indication, it’ll be there in a matter of years. Maybe we should have an early access component.