In the first comparison, you are not quantizing the activations (i.e the example tensor). When you call the torch.quantization.prepare and convert, the activations are quantized too. In this case, since you are skipping calibration (i.e you are calling convert after prepare), the activations are quantized with a default scale of 1.0, leading to a large quantization error. Please repeat this experiment with calibration and you should see lower error