Solved by raghuramank100 in post #3 
              
              
                FBGEMM is supported only for x86. You can get very good accuracies for qnnpack also. 
Please make sure that when you set: 
qconfig = torch.quantization.get_default_qconfig('qnnpack')

You also do: 
torch.backends.quantized.engine = 'qnnpack'

before running the model. 
The poorer accuracy numbers ar…