title: Polynomial Composition Activations: Unleashing the Dynamics of Large
    Language Models
summary: Transformers have found extensive applications across various domains due to
  the powerful fitting capabilities. This success can be partially attributed to
  their inherent nonlinearity. Thus, in addition to the ReLU function employed in
  the original transformer architecture, researchers have explored alternative
  modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment
  representational capacity. In this paper, we propose a novel category of
  polynomial composition activations (PolyCom), designed to optimize the dynamics
  of transformers. Theoretically, we provide a comprehensive mathematical
  analysis of PolyCom, highlighting its enhanced expressivity and efficacy
  relative to other activation functions. Notably, we demonstrate that networks
  incorporating PolyCom achieve the optimal approximation rate,
  indicating that PolyCom networks require minimal parameters to approximate
  general smooth functions in Sobolev spaces. We conduct empirical experiments on
  the pre-training configurations of large language models (LLMs), including both
  dense and sparse architectures. By substituting conventional activation
  functions with PolyCom, we enable LLMs to capture higher-order interactions
  within the data, thus improving performance metrics in terms of accuracy and
  convergence rates. Extensive experimental results demonstrate the effectiveness
  of our method, showing substantial improvements over other activation
  functions. Code is available at https://github.com/BryceZhuo/PolyCom.
paper_authors._id: 672cb0bfab349b67cffcc402
paper_authors.name: Zhijian Zhuo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T12:32:54.289Z
paper_authors._id: 66335b9c95c5b79ebf306f30
paper_authors.avatarUrl: /avatars/d57784ee65cbef014360c9bac1ad4119.svg
paper_authors.fullname: Zhijian Zhuo
paper_authors.type: user
paper_authors.user: BryceZhuo
paper_authors._id: 672cb0bfab349b67cffcc403
paper_authors.name: Ya Wang
paper_authors._id: 672cb0bfab349b67cffcc404
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:33:18.181Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 672cb0bfab349b67cffcc405
paper_authors.name: Xiaoqing Li
paper_authors._id: 672cb0bfab349b67cffcc406
paper_authors.name: Xun Zhou
paper_authors._id: 672cb0bfab349b67cffcc407
paper_authors.name: Jinwen Ma
paper_title: Polynomial Composition Activations: Unleashing the Dynamics of Large
    Language Models
paper_summary: Transformers have found extensive applications across various domains due to
  the powerful fitting capabilities. This success can be partially attributed to
  their inherent nonlinearity. Thus, in addition to the ReLU function employed in
  the original transformer architecture, researchers have explored alternative
  modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment
  representational capacity. In this paper, we propose a novel category of
  polynomial composition activations (PolyCom), designed to optimize the dynamics
  of transformers. Theoretically, we provide a comprehensive mathematical
  analysis of PolyCom, highlighting its enhanced expressivity and efficacy
  relative to other activation functions. Notably, we demonstrate that networks
  incorporating PolyCom achieve the optimal approximation rate,
  indicating that PolyCom networks require minimal parameters to approximate
  general smooth functions in Sobolev spaces. We conduct empirical experiments on
  the pre-training configurations of large language models (LLMs), including both
  dense and sparse architectures. By substituting conventional activation
  functions with PolyCom, we enable LLMs to capture higher-order interactions
  within the data, thus improving performance metrics in terms of accuracy and
  convergence rates. Extensive experimental results demonstrate the effectiveness
  of our method, showing substantial improvements over other activation
  functions. Code is available at https://github.com/BryceZhuo/PolyCom.
paper_ai_summary: Polynomial composition activations (PolyCom) enhance the representational capacity of transformer models, leading to improved approximation rates and performance in large language models.
paper_ai_keywords: transformers ReLU GeLU SwishGLU polynomial composition activations PolyCom optimal approximation rate Sobolev spaces large language models dense architectures sparse architectures accuracy convergence rates