title: Megalodon: Efficient LLM Pretraining and Inference with Unlimited
    Context Length
summary: The quadratic complexity and weak length extrapolation of Transformers limits
  their ability to scale to long sequences, and while sub-quadratic solutions
  like linear attention and state space models exist, they empirically
  underperform Transformers in pretraining efficiency and downstream task
  accuracy. We introduce Megalodon, a neural architecture for efficient sequence
  modeling with unlimited context length. Megalodon inherits the architecture of
  Mega (exponential moving average with gated attention), and further introduces
  multiple technical components to improve its capability and stability,
  including complex exponential moving average (CEMA), timestep normalization
  layer, normalized attention mechanism and pre-norm with two-hop residual
  configuration. In a controlled head-to-head comparison with Llama2, Megalodon
  achieves better efficiency than Transformer in the scale of 7 billion
  parameters and 2 trillion training tokens. Megalodon reaches a training loss of
  1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code:
  https://github.com/XuezheMax/megalodon
paper_authors._id: 661df6c69945e0962d7545ac
paper_authors.name: Xuezhe Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T06:50:43.101Z
paper_authors._id: 6624517c204cf7d22a90306e
paper_authors.avatarUrl: /avatars/82da7b7194f1d9ad49e8e1a470cf02ce.svg
paper_authors.fullname: Xuezhe Ma
paper_authors.type: user
paper_authors.user: maxma1987
paper_authors._id: 661df6c69945e0962d7545ad
paper_authors.name: Xiaomeng Yang
paper_authors._id: 661df6c69945e0962d7545ae
paper_authors.name: Wenhan Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:46:07.902Z
paper_authors._id: 66a159e608767ced9f37f49c
paper_authors.avatarUrl: /avatars/07d3ca897d0e1d6d467e220cd943ba3c.svg
paper_authors.fullname: Wenhan Xiong
paper_authors.type: user
paper_authors.user: xwhan
paper_authors._id: 661df6c69945e0962d7545af
paper_authors.name: Beidi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:36:05.317Z
paper_authors._id: 64b732f832403871593e082c
paper_authors.avatarUrl: /avatars/dd21932b0c167131ee7545a622c46c3c.svg
paper_authors.fullname: Beidi Chen
paper_authors.type: user
paper_authors.user: beidic
paper_authors._id: 661df6c69945e0962d7545b0
paper_authors.name: Lili Yu
paper_authors._id: 661df6c69945e0962d7545b1
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:14:53.490Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_authors._id: 661df6c69945e0962d7545b2
paper_authors.name: Jonathan May
paper_authors._id: 661df6c69945e0962d7545b3
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 661df6c69945e0962d7545b4
paper_authors.name: Omer Levy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:34:01.808Z
paper_authors._id: 610b71d7f5dd7bcefedaff55
paper_authors.avatarUrl: /avatars/b489c021e906010e661313da0b4aafab.svg
paper_authors.fullname: Omer Levy
paper_authors.type: user
paper_authors.user: omerlevy
paper_authors._id: 661df6c69945e0962d7545b5
paper_authors.name: Chunting Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:33:54.690Z
paper_authors._id: 64f8d07dd493d8b0d2c04ecd
paper_authors.avatarUrl: /avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Chunting
paper_title: Megalodon: Efficient LLM Pretraining and Inference with Unlimited
    Context Length
paper_summary: The quadratic complexity and weak length extrapolation of Transformers limits
  their ability to scale to long sequences, and while sub-quadratic solutions
  like linear attention and state space models exist, they empirically
  underperform Transformers in pretraining efficiency and downstream task
  accuracy. We introduce Megalodon, a neural architecture for efficient sequence
  modeling with unlimited context length. Megalodon inherits the architecture of
  Mega (exponential moving average with gated attention), and further introduces
  multiple technical components to improve its capability and stability,
  including complex exponential moving average (CEMA), timestep normalization
  layer, normalized attention mechanism and pre-norm with two-hop residual
  configuration. In a controlled head-to-head comparison with Llama2, Megalodon
  achieves better efficiency than Transformer in the scale of 7 billion
  parameters and 2 trillion training tokens. Megalodon reaches a training loss of
  1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code:
  https://github.com/XuezheMax/megalodon
paper_ai_keywords: quadratic complexity weak length extrapolation Transformers linear attention state space models Megalodon Neural architecture sequence modeling unlimited context length Mega exponential moving average with gated attention complex exponential moving average timestep normalization layer normalized attention mechanism pre-norm two-hop residual configuration Llama2 pretraining efficiency downstream task accuracy training loss