Expanded Gating Ranges Improve Activation Functions
<|reference_start|>Expanded Gating Ranges Improve Activation Functions: Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where the range of the gating function is between zero and one. In this paper, we explore the viability of using arctan as a gating mechanism. A self-gated activation function that uses arctan as its gating function has a monotonically increasing first derivative. To make this activation function competitive, it is necessary to introduce a trainable parameter for every MLP block to expand the range of the gating function beyond zero and one. We find that this technique also improves existing self-gated activation functions. We conduct an empirical evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and Expanded SiLU (xSiLU) and show that they outperform existing activation functions within a transformer architecture. Additionally, expanded gating ranges show promising results in improving first-order Gated Linear Units (GLU).<|reference_end|>
@article{huang2024expanded,
    title={Expanded Gating Ranges Improve Activation Functions},
    author={Allen Hao Huang},
    journal={arXiv preprint arXiv:2405.20768},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.20768},
    primaryClass={cs.NE cs.LG}
}