title: Scavenging Hyena: Distilling Transformers into Long Convolution Models
summary: The rapid evolution of Large Language Models (LLMs), epitomized by
  architectures like GPT-4, has reshaped the landscape of natural language
  processing. This paper introduces a pioneering approach to address the
  efficiency concerns associated with LLM pre-training, proposing the use of
  knowledge distillation for cross-architecture transfer. Leveraging insights
  from the efficient Hyena mechanism, our method replaces attention heads in
  transformer models by Hyena, offering a cost-effective alternative to
  traditional pre-training while confronting the challenge of processing long
  contextual information, inherent in quadratic attention mechanisms. Unlike
  conventional compression-focused methods, our technique not only enhances
  inference speed but also surpasses pre-training in terms of both accuracy and
  efficiency. In the era of evolving LLMs, our work contributes to the pursuit of
  sustainable AI solutions, striking a balance between computational power and
  environmental impact.
paper_authors._id: 65bb04827677ff48f5c4be6d
paper_authors.name: Tokiniaina Raharison Ralambomihanta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:01:33.110Z
paper_authors._id: 65bc13bb4e82d4524345847e
paper_authors.avatarUrl: /avatars/fdcae0cbd7da356d6cd6d49be1af85ed.svg
paper_authors.fullname: Toky Raharison Ralambomihanta
paper_authors.type: user
paper_authors.user: JustToky
paper_authors._id: 65bb04827677ff48f5c4be6e
paper_authors.name: Shahrad Mohammadzadeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:01:40.347Z
paper_authors._id: 659d84dba74945e564da065b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/659d84dba74945e564da065b/NMavmXi1gkmtKO0GI-HKX.png
paper_authors.fullname: MZ
paper_authors.type: user
paper_authors.user: Shahradmz
paper_authors._id: 65bb04827677ff48f5c4be6f
paper_authors.name: Mohammad Sami Nur Islam
paper_authors._id: 65bb04827677ff48f5c4be70
paper_authors.name: Wassim Jabbour
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:01:35.222Z
paper_authors._id: 65bc0291b7db0ab095d59735
paper_authors.avatarUrl: /avatars/d9736dc422cbc4be86fa56547b181caa.svg
paper_authors.fullname: Wassim Jabbour
paper_authors.type: user
paper_authors.user: WassimJabz
paper_authors._id: 65bb04827677ff48f5c4be71
paper_authors.name: Laurence Liang
paper_title: Scavenging Hyena: Distilling Transformers into Long Convolution Models
paper_summary: The rapid evolution of Large Language Models (LLMs), epitomized by
  architectures like GPT-4, has reshaped the landscape of natural language
  processing. This paper introduces a pioneering approach to address the
  efficiency concerns associated with LLM pre-training, proposing the use of
  knowledge distillation for cross-architecture transfer. Leveraging insights
  from the efficient Hyena mechanism, our method replaces attention heads in
  transformer models by Hyena, offering a cost-effective alternative to
  traditional pre-training while confronting the challenge of processing long
  contextual information, inherent in quadratic attention mechanisms. Unlike
  conventional compression-focused methods, our technique not only enhances
  inference speed but also surpasses pre-training in terms of both accuracy and
  efficiency. In the era of evolving LLMs, our work contributes to the pursuit of
  sustainable AI solutions, striking a balance between computational power and
  environmental impact.
paper_ai_summary: Knowledge distillation using the Hyena mechanism enhances both accuracy and efficiency in LLM pre-training, replacing attention heads and addressing long contextual information processing.
paper_ai_keywords: Large Language Models LLM GPT-4 knowledge distillation Hyena mechanism transformer models attention heads computational power environmental impact