right. that is a very helpful visualization. I see what is happening and it is very clear to me that the gradients don’t make it past the backward call. I still don’t see why they dont propagate through the parameter update though.

The parameters require grad
The loss required grad
I set retain_graph and create_graph

I manually update the parameters (without calling no_grad) and I also don’t use the in-place operations that are in the optimizers which avoid the differentiation graph (AFAIK)

This chain of events leads me to believe that the graph should still be stored and the top graph you posted and the bottom graph should be connected from MeanBackward0 to TBackward with the inputs coming in separately.
This is a common pattern in meta learning if I understand it correctly and I don’t see why the graphs should be separated given my bullet points above.