This thread says exporting quantized pytorch models to onnx is not yet supported.

  
    
    
    ONNX export of quantized model quantization
  
  
    General export of quantized models to ONNX isn’t currently supported. We only support conversion to ONNX for Caffe2 backend. This thread has additional context on what we currently support - ONNX export of quantized model
  


You can try quantizing after you export pytorch model to onnx by using onnxruntime 49.
Otherwise, you may want to check out if direct export from pytorch to tensorrt supports quantized models.

  
      
      GitHub
  
  
    

NVIDIA-AI-IOT/torch2trt 131

An easy to use PyTorch to TensorRT converter. Contribute to NVIDIA-AI-IOT/torch2trt development by creating an account on GitHub.