Hi,
I’m in trouble with the same case.
No speed-up.
I ran the bert tutorial in a conda env on iMac.
FP32:  437.98 MB,  106.6 sec,  acc&f1 0.8811
QINT8:  181.44 MB, 108.2 sec, acc&f1 0.8799
python -V
3.8.5

torch.__version__ 
1.6.0

<conda info>

    active environment : quant_bert
    active env location : /Users/*******/anaconda/envs/quant_bert
            shell level : 1
       user config file : /Users/*******/.condarc
 populated config files :
          conda version : 4.5.11
    conda-build version : 2.0.2
         python version : 3.5.6.final.0
       base environment : /Users/*******/anaconda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/osx-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /Users/*******/anaconda/pkgs
                          /Users/*******/.conda/pkgs
       envs directories : /Users/*******/anaconda/envs
                          /Users/*******/.conda/envs
               platform : osx-64
             user-agent : conda/4.5.11 requests/2.14.2 CPython/3.5.6 Darwin/19.6.0 OSX/10.15.6

<H/W spec.>
macOS Catalina 10.15.6
iMac (Retina 5K, 27-inch, Late 2014) 
Quad-Core Intel Core i5, 3.5GHz
32GB 1600 MHz DDR3
AMD Radeon R9 M290X 2 GB
L2 cache: 256 KB / core
L3 cache: 6 MB

Thanks.