The scale and zero_point aren’t trained - they are calculated by observers inserted in the network. You can implement an observer specific to your use-case which will fix the zero_point at 0. For reference the zero_point calculation happens in https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py#L187 5
Observers are set when you initialize the qconfig (in this case you seem to be using the default. i.e. https://github.com/pytorch/pytorch/blob/master/torch/quantization/qconfig.py#L90 2