OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models
<|reference_start|>OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models: To overcome the burden on the memory size and bandwidth due to ever-increasing size of large language models (LLMs), aggressive weight quantization has been recently studied, while lacking research on quantizing activations. In this paper, we present a hardware-software co-design method that results in an energy-efficient LLM accelerator, named OPAL, for generation tasks. First of all, a novel activation quantization method that leverages the microscaling data format while preserving several outliers per sub-tensor block (e.g., four out of 128 elements) is proposed. Second, on top of preserving outliers, mixed precision is utilized that sets 5-bit for inputs to sensitive layers in the decoder block of an LLM, while keeping inputs to less sensitive layers to 3-bit. Finally, we present the OPAL hardware architecture that consists of FP units for handling outliers and vectorized INT multipliers for dominant non-outlier related operations. In addition, OPAL uses log2-based approximation on softmax operations that only requires shift and subtraction to maximize power efficiency. As a result, we are able to improve the energy efficiency by 1.6~2.2x, and reduce the area by 2.4~3.1x with negligible accuracy loss, i.e., <1 perplexity increase.<|reference_end|>
@article{koo2024opal:,
    title={OPAL: Outlier-Preserved Microscaling Quantization Accelerator for
  Generative Large Language Models},
    author={Jahyun Koo, Dahoon Park, Sangwoo Jung, Jaeha Kung},
    journal={arXiv preprint arXiv:2409.05902},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05902},
    primaryClass={cs.LG cs.AR cs.CL}
}