Solved by hidefromkgb in post #2 
              
              
                Okay, now that I know more about PT I can answer my own question. 
The solution is to use TorchScript and work directly with the traced graph. 
For that you`ll need a torch::jit backend. 
Here`s a useful article on the topic. Bear in mind however that it`s already quite outdated, so e.g. CustomFuseG…