The scale and zero point for input are calculated after a calibration step across a representative set of images. By default, for PyTorch quantization, these are chosen such that quantized values are in the range [0, 127] for activations and zero_point is somewhere in the same range (e.g., 57 for the model here). We restrict quantized activation values to [0, 127] instead of [0, 255] for uint8 to avoid the saturation explained here: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qconv.cpp#L633-L661 16
When you force the zero_point to 128, the quantized activations become in the range [0, 255] and it is possible that the saturation happens. BTW, any reason for forcing zero_point to a large value? Let the calibration step pick scale and zero_points for the model you are quantizing.
It is strange that you see different behavior on the cpus you mentioned. I would expect the same behavior on all the CPUs you mentioned.