The issue is that default pytorch uses fbgemm as a backend for quantized operators. You can check this by
print(torch.backends.quantized.engine)

Caffe2 int8conv uses the qnnpack engine by default. To ensure that pytorch and c2 match, you will need to do the following:
Once I do that the outputs match.

model.qconfig = torch.quantization.get_default_qconfig("qnnpack")
torch.backends.quantized.engine = 'qnnpack'
# Set engine to qnnpack
print(model.qconfig)
model = torch.quantization.prepare(model, inplace=True);