Surya_J:

    def forward(self, X):
        # Outputs are dequantized
        if self.quantize == True:
            output_out = self.dequant(output_out)
        
        # pass through other layers
        
        # Outputs are dequantized
        if self.quantize == True:
            output_out = self.dequant(output_out)
        return output_out



It might be a typo, but it should be something like
# quantize inputs (currently your code doesn't have this in what you pasted)
# run quantized model
# dequantize outputs

Other than that, your code looks right, as long as your calibration dataset is representative of your inference dataset.  Are you sure the expected output from your input is not close to a matrix of zeros?  Do you get the same thing for other outputs?
You could also try using PyTorch Numeric Suite Tutorial — PyTorch Tutorials 1.9.1+cu102 documentation 3 to see if you can bisect the difference to a specific layer.