Yeah, I’ll let @Michael_Suo elaborate if he wants, but in my understanding:

Converting models between 2 frameworks is hard. Each framework needs to support the same operations and the conversion process introduces opportunities for bugs
Converting models between 3 frameworks (PT --> ONNX --> TrT) introduces more opportunities for failures or incompatibilities
PyTorch core developers are focused on JIT and will not invest significant effort in ONNX

Today, based on my reading and testing, going from PyTorch --> ONNX --> TensorRT will be easier for some use cases, but I think that will change in the future, and the community should invest in JIT --> TensorRT conversion directly.
The torch2trt repo below works well for simple / popular models like ResNet, which makes it possible to deploy hybrid models of JIT + TrT (most object detectors will work like this). But it’s tracing-based, unlike ONNX --> TrT which does direct graph parsing, so doing a proper port of a complex architecture would require some surgery.

  
      
      GitHub
  
  
    

NVIDIA-AI-IOT/torch2trt 25

An easy to use PyTorch to TensorRT converter. Contribute to NVIDIA-AI-IOT/torch2trt development by creating an account on GitHub.


  
  
    
    
  
  


Anyway, I hope the PyTorch/Nvidia developers will consider collaborating on the Torch2Trt repo to make it better in the future. In my performance testing, TensorRT is at least 2x faster than raw JIT (I don’t see any speedups for JIT over raw PyTorch for any architecture except a tiny benefit from c++ runtime) for architectures like ResNet, however the hybrid models (ResNet backbone in TrT, Object Detector head in JIT), can close that gap considerably.
I’d like to see PyTorch developers invest more in JIT performance, interoperability with 3P accelerators, detailed documentation and examples of how to squeeze out model performance with JIT, and provide some benchmarks on the various combinations.