Here I have more details…
Following is the model output for some inputs. The LHS bracket is target value. Values in RHS are output values for 2 channels (background and foreground). The RHS values are passed to a Softmax function and the final result is obtained (…and to be compared with target)
Model output before quantization:
 ..........(1.0) (1.42 16.16)
 ..........(1.0) (-40.55 42.14)
 ..........(0.0) (15.20 -19.15)
 ..........(1.0) (-21.16 25.58)
 ..........(1.0) (-43.54 41.77)
 ..........(0.0) (19.74 -23.29)
 ..........(1.0) (-29.66 33.56)
 ..........(1.0) (1.23 -7.96)
 ..........(1.0) (-35.54 42.13)
 ..........(0.0) (16.74 -19.38)
 ..........(0.0) (9.40 -2.54)
 ..........(0.0) (21.67 -27.59)
 ..........(1.0) (-52.96 53.53)
 ..........(0.0) (18.02 -20.90)
 ..........(1.0) (-19.79 22.51)
 ..........(0.0) (13.33 -20.11)
 ..........(0.0) (29.95 -31.26)
 ..........(0.0) (23.35 -29.38)
 ..........(1.0) (-15.23 9.97)
 ..........(0.0) (18.14 -24.80)
 ..........(0.0) (19.13 -26.98)
 ..........(1.0) (-18.12 22.96)

Model output after FBGEMM quantization - as you can see below, the output values did change, but only to small extent
..........(1.0) (0.00 18.41)
 ..........(1.0) (-45.41 50.32)
 ..........(0.0) (13.50 -14.73)
 ..........(1.0) (-24.55 30.69)
 ..........(1.0) (-39.28 38.05)
 ..........(0.0) (13.50 -17.18)
 ..........(1.0) (-22.09 25.78)
 ..........(1.0) (2.45 -7.36)
 ..........(1.0) (-23.32 29.46)
 ..........(0.0) (17.18 -23.32)
 ..........(0.0) (12.27 -6.14)
 ..........(0.0) (20.87 -23.32)
 ..........(1.0) (-45.41 49.10)
 ..........(0.0) (15.96 -18.41)
 ..........(1.0) (-17.18 20.87)
 ..........(0.0) (11.05 -18.41)
 ..........(0.0) (27.00 -27.00)
 ..........(0.0) (17.18 -23.32)
 ..........(1.0) (-2.45 1.23)
 ..........(0.0) (15.96 -20.87)
 ..........(0.0) (18.41 -24.55)
 ..........(1.0) (-15.96 20.87)

Now look at following model output for QNNPACK quantization. The output is very different from the unquantized version. In particular, values for all pixels is positive for background channel and negative for foreground channel.
..........(1.0) (14.06 -17.12)
 ..........(1.0) (11.61 -16.51)
 ..........(0.0) (20.17 -25.06)
 ..........(1.0) (18.34 -22.01)
 ..........(1.0) (15.89 -14.06)
 ..........(0.0) (20.17 -25.67)
 ..........(1.0) (22.62 -29.34)
 ..........(1.0) (24.45 -28.73)
 ..........(1.0) (14.06 -20.17)
 ..........(0.0) (22.62 -28.12)
 ..........(0.0) (27.51 -20.17)
 ..........(0.0) (21.40 -23.84)
 ..........(1.0) (20.78 -29.34)
 ..........(0.0) (17.12 -23.23)
 ..........(1.0) (28.73 -31.18)
 ..........(0.0) (18.34 -20.78)
 ..........(0.0) (20.17 -23.84)
 ..........(0.0) (20.78 -23.84)
 ..........(1.0) (13.45 -16.51)
 ..........(0.0) (17.12 -21.40)
 ..........(0.0) (21.40 -25.67)
 ..........(1.0) (21.40 -26.90)

Any thoughts by anyone?