Polynomial Composition Activations: Unleashing the Dynamics of Large
  Language Models
Transformers have found extensive applications across various domains due to
the powerful fitting capabilities. This success can be partially attributed to
their inherent nonlinearity. Thus, in addition to the ReLU function employed in
the original transformer architecture, researchers have explored alternative
modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment
representational capacity. In this paper, we propose a novel category of
polynomial composition activations (PolyCom), designed to optimize the dynamics
of transformers. Theoretically, we provide a comprehensive mathematical
analysis of PolyCom, highlighting its enhanced expressivity and efficacy
relative to other activation functions. Notably, we demonstrate that networks
incorporating PolyCom achieve the optimal approximation rate,
indicating that PolyCom networks require minimal parameters to approximate
general smooth functions in Sobolev spaces. We conduct empirical experiments on
the pre-training configurations of large language models (LLMs), including both
dense and sparse architectures. By substituting conventional activation
functions with PolyCom, we enable LLMs to capture higher-order interactions
within the data, thus improving performance metrics in terms of accuracy and
convergence rates. Extensive experimental results demonstrate the effectiveness
of our method, showing substantial improvements over other activation
functions. Code is available at https://github.com/BryceZhuo/PolyCom.
672cb0bfab349b67cffcc402
Zhijian Zhuo
claimed_verified
2025-03-24T12:32:54.289Z
66335b9c95c5b79ebf306f30
/avatars/d57784ee65cbef014360c9bac1ad4119.svg
Zhijian Zhuo
user
BryceZhuo
672cb0bfab349b67cffcc403
Ya Wang
672cb0bfab349b67cffcc404
Yutao Zeng
claimed_verified
2024-11-15T09:33:18.181Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
672cb0bfab349b67cffcc405
Xiaoqing Li
672cb0bfab349b67cffcc406
Xun Zhou
672cb0bfab349b67cffcc407
Jinwen Ma
Polynomial Composition Activations: Unleashing the Dynamics of Large
  Language Models
Transformers have found extensive applications across various domains due to
the powerful fitting capabilities. This success can be partially attributed to
their inherent nonlinearity. Thus, in addition to the ReLU function employed in
the original transformer architecture, researchers have explored alternative
modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment
representational capacity. In this paper, we propose a novel category of
polynomial composition activations (PolyCom), designed to optimize the dynamics
of transformers. Theoretically, we provide a comprehensive mathematical
analysis of PolyCom, highlighting its enhanced expressivity and efficacy
relative to other activation functions. Notably, we demonstrate that networks
incorporating PolyCom achieve the optimal approximation rate,
indicating that PolyCom networks require minimal parameters to approximate
general smooth functions in Sobolev spaces. We conduct empirical experiments on
the pre-training configurations of large language models (LLMs), including both
dense and sparse architectures. By substituting conventional activation
functions with PolyCom, we enable LLMs to capture higher-order interactions
within the data, thus improving performance metrics in terms of accuracy and
convergence rates. Extensive experimental results demonstrate the effectiveness
of our method, showing substantial improvements over other activation
functions. Code is available at https://github.com/BryceZhuo/PolyCom.
Polynomial composition activations (PolyCom) enhance the representational capacity of transformer models, leading to improved approximation rates and performance in large language models.
transformers ReLU GeLU SwishGLU polynomial composition activations PolyCom optimal approximation rate Sobolev spaces large language models dense architectures sparse architectures accuracy convergence rates