Hi @Michael_Suo,
Curious if you could elaborate on how “dynamic” plays into this — if a compiled TorchScript model has been through profile-guided optimization and had all of the control flow stripped out, the actual autograd graph structure should be the same at each inference pass, yes?
When I run autograd with create_graph = True, is a graph being created that PyTorch knows how to execute, or only how to differentiate further?
The motivation for these questions is models whose inference pass involves a gradient of their initial prediction, and the hope that it might be possible to save/cache the compute graph representing their gradient as a model in its own right.
Thanks for your help!