In general, what percentage of forward-backward combined time is spent reforming the computation graph? It seems like a trivial optimization to simply save whatever graph object is formed between forward passes if the underlying computation graph is in fact static. If this is possible, there’s no reason why static computation graph approaches need lazy execution (which is what makes Tensorflow a pain). For a static graph, the computation graph could be formed on the first forward pass (no lazy execution) and then simply saved. I feel like few applications actually use dynamic computation graphs (i.e. where different functions are run in different situations) and instead are all essentially static graphs that could benefit from this.