ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
<|reference_start|>ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs: Sparse computation offers a compelling solution for the inference of Large Language Models (LLMs) in low-resource scenarios by dynamically skipping the computation of inactive neurons. While traditional approaches focus on ReLU-based LLMs, leveraging zeros in activation values, we broaden the scope of sparse LLMs beyond zero activation values. We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold, demonstrating that non-ReLU LLMs also exhibit sparse activation. To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$^2$. The results indicate that models employing ReLU$^2$ excel across all three evaluation aspects, highlighting its potential as an efficient activation function for sparse LLMs. We will release the code to facilitate future research.<|reference_end|>
@article{zhang2024relu$^2$,
    title={ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse
  LLMs},
    author={Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun
  Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, Maosong Sun},
    journal={arXiv preprint arXiv:2402.03804},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03804},
    primaryClass={cs.LG cs.AI}
}