It could be that your model is sensitive to quantization.  There is a prototype tool to help narrow this down to a particular layer: PyTorch Numeric Suite Tutorial — PyTorch Tutorials 1.7.1 documentation 3 .  One thing to try could be to run an example input through this tool and see if there is a particular problematic layer where things diverge.