I’m just referring to most typical feedforward ML models (like many models in computer vision, e.g. AlexNet, ResNet) where the computation graphs are set by the architecture. I see your point though (although I feel like even recurrent models don’t necessarily require full dynamic computation graph tracking and could just “copy-and-paste” whatever computation graph building operation occurs for one recursion for each recursion). Full disclosure, I’m not terribly familiar with the implementation of autograd.
The big question is how much of a performance hit is given by dynamic graph construction.