you can quantize the model. As mentioned in the PyTorch doc 7

PyTorch supports INT8 quantization compared to typical FP32 models allowing for a 4x reduction in the model size and a 4x reduction in memory bandwidth requirements. Hardware support for INT8 computations is typically 2 to 4 times faster compared to FP32 compute.

this notebook 39 shows the benchmarks of using quantized models with onnx.
download1306×990 20.1 KB