OK, here is maybe a better explanation of what I try to understand:
When I use some pre-fabricated model for my network, the normal process of training it is

forward the input through the net and get the output
calculate the loss
zero the gradients (through optimizer.zero_grad() or mynetword.zero_grad())
backward the loss using something like loss.backward()

take an optimizer step
rinse and repeat

Now, nowhere have I ever seen that in the loss.backward() step we would specify retain_graph=True yet this always works!? In order for this to work, the loss.backward() function has to recursively call somevariable.backward() on all of the parameters of its network but will these calls use the retain_graph option? I assume not. So why does this work in general, but not in my case? Are there rules when it will work and when not?
I mainly want to understand what is going on and how PyTorch works so I do not even know yet if I “like” retaining the graph 
But from the fact that practically all examples do not retain the graph and still work, my feeling is that there must be an advantage to not retaining it … and I would like to understand that as well!