Hi,
I don’t think this is what he wants.
retain_graph can be used if you want to call backward() multiple times on the same graph.
create_graph is used if you want to backward() through the backward(). But since this second backward() will go through the current graph, then retain_graph has to be set as well. If you set it to False, you will get an error.
@kolorado the graph is deleted as soon as all references to it are gone. This would be the Tensor that contains the loss for example.
Note that pytorch uses a custom gpu memory allocator. So when objects are freed, they are not returned to the OS directly. And so you won’t see in nvidia-smi that objects have been freed. This memory is available to be reused by pytorch though. So the next forward will actually not allocate more memory on your GPU. You can find the functions here in the doc 21 to be able to see exactly how much memory is used and how much is cached to be re-used later.