I think your bullet points are correct.
Also,



 deltaskelta:

This chain of events leads me to believe that the graph should still be stored


would be true.
However, MeanBackward0 and TBackward will not connect the graphs.
If these graphs would be connected, then also this dummy code:
lin = nn.Linear(1, 1)

x1 = torch.randn(1, 1)
out1 = lin(x1)

x2 = torch.randn(1, 1)
out2 = lin(x2)

would create a single graph, which is not the case.



 deltaskelta:

This is a common pattern in meta learning if I understand it correctly and I don’t see why the graphs should be separated given my bullet points above.


I have the feeling, that I’m missing something, as I’m not deeply familiar with meta learning.
CC @albanD to see what I’m missing and correct me.