Hi Folks,
A few comments and pointers. Caffe2 is unsupported at this point so, if you are going from PyTorch to ONNX, you can use the exporter directly (which is well supported by MSFT). A tutorial you can use is here: https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html 53
For TensorRT, with the improvements in the ONNX exporter, the experience should be better now. We can ping Nvidia folks if there are any issues: https://github.com/onnx/onnx-tensorrt 53
Generally speaking, for backend performance libraries (relating to the QNNPACK comment above), the following is guidance depending on the target platform:
Server CPU (x86) - MKL/DNN and FBGEMM(8bit integer quantized)
ARM/Neon CPU - QNNPACK (8b integer quantized, ready today) and XNNPACK (current WIP for integration)
Hope this helps! Happy NY!!