Usually after a backpropagation you process the next batch so you don’t need the gradients of the previous batch anymore.
Besides debugging I don’t have any scenario for which I need to backprop twice through the same operation graph, I’m sure there are though.
The default is to drop the variables/gradient to save on memory.
All the source code of PyTorch is on github.  From the name, I guess that self._execution_engine.run_backward is traversing the operation graph in reverse, calling backward if defined, or using the autograd if not.
Each node in the graph have several properties that are defined in the autograd folder of PyTorch