Training-Free Activation Sparsity in Large Language Models
<|reference_start|>Training-Free Activation Sparsity in Large Language Models: Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass. However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL, a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53$\times$ and 1.8$\times$ at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.<|reference_end|>
@article{liu2024training-free,
    title={Training-Free Activation Sparsity in Large Language Models},
    author={James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, Ben
  Athiwaratkun},
    journal={arXiv preprint arXiv:2408.14690},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.14690},
    primaryClass={cs.CL cs.AI}
}