focused on the weight quantization of generative models by applying distillation based
on generations produced by the pre-trained model. Frantar et al. [133] proposed a one-
shot weight quantization method based on approximated second-order information,
reducing the bandwidth of the GPT model down to 3 or 4 bits per weight. Some works
achieve weight quantization during training. For example, Yang et al. [134] proposed
Beyond Efficiency