FBGEMM is supported only for x86. You can get very good accuracies for qnnpack also.
Please make sure that when you set:
qconfig = torch.quantization.get_default_qconfig('qnnpack')

You also do:
torch.backends.quantized.engine = 'qnnpack'

before running the model.
The poorer accuracy numbers are likely due to FBGEMM saturating for large weight/activation values, due to this issue:

  
      github.com
  
  
    intel/mkl-dnn/blob/f38fecf5b76421fe277cfb15ec1d5090f1d30c07/doc/advanced/int8_computations.md#1-inputs-of-mixed-type-u8-and-s8 12
Int8 Computation Aspects {#dev_guide_int8_computations}
=======================================================

> This document uses **int8** to denote 8-bit integer no matter whether it is
> signed or unsigned. To emphasize the signedness of the data type
> **u8** (`uint8_t`) or **s8** (`int8_t`) are used. In particular, if a
> primitive has two inputs the types would be written using "/". For instance:
> - int8 GEMM denotes any integer GEMM with 8-bit integer inputs, while
> - u8/s8 GEMM denotes dnnl_gemm_u8s8s32() only.

The operation primitives that work with the int8 data type
(#dnnl::memory::data_type::s8 and #dnnl::memory::data_type::u8)
typically use s32 (`int32_t`) as an intermediate data type
(#dnnl::memory::data_type::s32) to avoid integer overflows.

For instance, the int8 average [pooling](@ref dev_guide_pooling) primitive
accumulates the int8 input values in a window to an s32 accumulator, then
divides the result by the window size, and then stores the result back to the
int8 destination:



  This file has been truncated. show original