If you can split your model in quantized and fp32 parts. Then you can try to use NNAPI workflow to lower quantized part. (maybe even fp32). Have you tried that? (Beta) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.10.1+cu102 documentation 1