Thanks for your reply @albanD!
Indeed you are correct this is the function I had in mind.
From reading the documentation I did not understand clearly its puprose (maybe an example would have helped)
I just tried to apply my mental flow with it and it works. The defaults do not allow for an easy try and learn though since they do not retain the graph:
>>> w = torch.tensor([1., 1., 1.], requires_grad=True)
>>> func = w[0] ** 2 + 2 * w[1] + w[2]
>>> oups_had_forgotten = func + 3 * w[1]
>>> torch.autograd.grad(oups_had_forgotten, w)
(tensor([2., 5., 1.]),)
>>> torch.autograd.grad(func, w)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\remi\.virtualenvs\zerotogan--y2dYRx8\lib\site-packages\torch\autograd\__init__.py", line 158, in grad
    inputs, allow_unused)
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Is the aggressive freeing of buffers a performance tweek?
But if I choose to retain the graph at each step then I can at least do some retry:
>>> w = torch.tensor([1., 1., 1.], requires_grad=True)
>>> func = w[0] ** 2 + 2 * w[1] + w[2]
>>> oups_had_forgotten = func + 3 * w[1]
>>> torch.autograd.grad(func, w, retain_graph=True)
(tensor([2., 2., 1.]),)
>>> torch.autograd.grad(oups_had_forgotten, w, retain_graph=True)
(tensor([2., 5., 1.]),)

Thanks !