riley:

It seems like a trivial optimization to simply save whatever graph object is formed between forward passes if the underlying computation graph is in fact static.


This was basically done using tracing. However, the current way is to use torch.jit.script, as it will also track the control flow.



 riley:

I feel like few applications actually use dynamic computation graphs (i.e. where different functions are run in different situations) and instead are all essentially static graphs that could benefit from this.


Really? I have the feeling that especially due to the growth of NLP models the control flow capturing is getting even more important.
Which models have you compared from which domain?