Megalodon: Efficient LLM Pretraining and Inference with Unlimited
  Context Length
The quadratic complexity and weak length extrapolation of Transformers limits
their ability to scale to long sequences, and while sub-quadratic solutions
like linear attention and state space models exist, they empirically
underperform Transformers in pretraining efficiency and downstream task
accuracy. We introduce Megalodon, a neural architecture for efficient sequence
modeling with unlimited context length. Megalodon inherits the architecture of
Mega (exponential moving average with gated attention), and further introduces
multiple technical components to improve its capability and stability,
including complex exponential moving average (CEMA), timestep normalization
layer, normalized attention mechanism and pre-norm with two-hop residual
configuration. In a controlled head-to-head comparison with Llama2, Megalodon
achieves better efficiency than Transformer in the scale of 7 billion
parameters and 2 trillion training tokens. Megalodon reaches a training loss of
1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code:
https://github.com/XuezheMax/megalodon
661df6c69945e0962d7545ac
Xuezhe Ma
claimed_verified
2024-04-22T06:50:43.101Z
6624517c204cf7d22a90306e
/avatars/82da7b7194f1d9ad49e8e1a470cf02ce.svg
Xuezhe Ma
user
maxma1987
661df6c69945e0962d7545ad
Xiaomeng Yang
661df6c69945e0962d7545ae
Wenhan Xiong
admin_assigned
2024-12-16T13:46:07.902Z
66a159e608767ced9f37f49c
/avatars/07d3ca897d0e1d6d467e220cd943ba3c.svg
Wenhan Xiong
user
xwhan
661df6c69945e0962d7545af
Beidi Chen
admin_assigned
2024-04-18T09:36:05.317Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
661df6c69945e0962d7545b0
Lili Yu
661df6c69945e0962d7545b1
Hao Zhang
claimed_verified
2025-01-01T20:14:53.490Z
62d363143eebd640a4fa41fa
https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
Hao Zhang
user
zhisbug
661df6c69945e0962d7545b2
Jonathan May
661df6c69945e0962d7545b3
Luke Zettlemoyer
661df6c69945e0962d7545b4
Omer Levy
admin_assigned
2024-04-18T09:34:01.808Z
610b71d7f5dd7bcefedaff55
/avatars/b489c021e906010e661313da0b4aafab.svg
Omer Levy
user
omerlevy
661df6c69945e0962d7545b5
Chunting Zhou
admin_assigned
2024-04-18T09:33:54.690Z
64f8d07dd493d8b0d2c04ecd
/avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
Zhou
user
Chunting
Megalodon: Efficient LLM Pretraining and Inference with Unlimited
  Context Length
The quadratic complexity and weak length extrapolation of Transformers limits
their ability to scale to long sequences, and while sub-quadratic solutions
like linear attention and state space models exist, they empirically
underperform Transformers in pretraining efficiency and downstream task
accuracy. We introduce Megalodon, a neural architecture for efficient sequence
modeling with unlimited context length. Megalodon inherits the architecture of
Mega (exponential moving average with gated attention), and further introduces
multiple technical components to improve its capability and stability,
including complex exponential moving average (CEMA), timestep normalization
layer, normalized attention mechanism and pre-norm with two-hop residual
configuration. In a controlled head-to-head comparison with Llama2, Megalodon
achieves better efficiency than Transformer in the scale of 7 billion
parameters and 2 trillion training tokens. Megalodon reaches a training loss of
1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code:
https://github.com/XuezheMax/megalodon
Megaldon, an enhanced neural architecture, provides efficient long-sequence modeling by combining and improving upon components from mega and transformer models, achieving better pretraining efficiency and performance.
quadratic complexity weak length extrapolation linear attention state space models megalodon exponential moving average gated attention complex exponential moving average timestep normalization layer normalized attention mechanism pre-norm two-hop residual configuration llama2