You may want to try some quantization accuracy improvement techniques such as
per channel quantization for weights
Quantization aware training
Measuring torch.norm between float model and quantize model to see where it’s off the most.