I am using Unet model for semantic segmentation. I pass a batch of images to the model. The model is expected to output 0 or 1 for each pixel of the image (depending upon whether pixel is part of person object or not). 0 is for background, and 1 is for foreground.
I am trying to quantize the Unet model with Pytorch quantization apis (static quantization). The model accuracy is good for FBGEMM config. However, the model outputs all pixels as black pixels (background pixels) for all images. The model output is very high positive for background channel and very high negative for forground channel. (I perform softmax to get final classification of the pixel). The code works fine for FBGEMM. Same code produces very different results for QNNPACK. Is there anything missing for QNNPACK code? I am pasting my code below for reference.
# Static Quantization - FBGEMM/QNNPACK

import torch.quantization as Q
import torch

framework = 'qnnpack'   # change to fbgemm for x86 architecture
per_channel_quantized_model = model
per_channel_quantized_model.eval()
per_channel_quantized_model.fuse_model()
per_channel_quantized_model.qconfig = Q.get_default_qconfig(framework)
torch.backends.quantized.engine = framework

print("Preparing . . .")
Q.prepare(per_channel_quantized_model, inplace=True)

print("Running model . . .")
eval_model_for_quantization(per_channel_quantized_model, 'cpu')

print("Converting . . .")
Q.convert(per_channel_quantized_model, inplace=True)

print("***************")
print(per_channel_quantized_model)

print("***************")
print("Checking Accuracy . . .")
accuracy = eval_model_for_quantization(per_channel_quantized_model, 'cpu')
print()
print('Evaluation accuracy after quantization', accuracy)

One issue that I did encounter is that the upsampling layers of Unet use nn.ConvTranspose2d which is not supported for quantization. Hence before this layer, we need to dequantize tensors, apply nn.ConvTranspose2d, and then requantize for subsequent layers. Can this be reason for lower accuracy?
NOTE -  I did try with QAT for QNNPACK. However, model output does not change i.e. it gives out all black pixels.