I directly add X = X.float() above where the error appears in torch.quantization.fake_quantize.py.  Because I found the dtype  of X is torch.float16. Surprisingly, it works, the model can be successfully trained.
        if self.fake_quant_enabled[0] == 1:
            if self.qscheme == torch.per_channel_symmetric or self.qscheme == torch.per_channel_affine:
                X = torch.fake_quantize_per_channel_affine(X, self.scale, self.zero_point,
                                                           self.ch_axis, self.quant_min, self.quant_max)
            else:
                import pdb
                # try:
                # I modified the code here. $$$$$$$$$$$$$$$$$
                X= X.float()
                X = torch.fake_quantize_per_tensor_affine(X, float(self.scale),
                                                          int(self.zero_point), self.quant_min,
                                                          self.quant_max)
                # except:
                #     pdb.set_trace()
        return X

Is this normal?
I haven’t completed training now, thus I can’t tell whether I can get a useful model. My PyTorch version is 1.7.0