title: Griffin: Mixing Gated Linear Recurrences with Local Attention for
    Efficient Language Models
summary: Recurrent neural networks (RNNs) have fast inference and scale efficiently on
  long sequences, but they are difficult to train and hard to scale. We propose
  Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that
  mixes gated linear recurrences with local attention. Hawk exceeds the reported
  performance of Mamba on downstream tasks, while Griffin matches the performance
  of Llama-2 despite being trained on over 6 times fewer tokens. We also show
  that Griffin can extrapolate on sequences significantly longer than those seen
  during training. Our models match the hardware efficiency of Transformers
  during training, and during inference they have lower latency and significantly
  higher throughput. We scale Griffin up to 14B parameters, and explain how to
  shard our models for efficient distributed training.
paper_authors._id: 65e155a99944a44329e45362
paper_authors.name: Soham De
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T13:50:27.725Z
paper_authors._id: 661415d9f9e58fe02bd8661d
paper_authors.avatarUrl: /avatars/7a28bb507b8bac72a8b161e774b19fbd.svg
paper_authors.fullname: Soham De
paper_authors.type: user
paper_authors.user: sohamde
paper_authors._id: 65e155a99944a44329e45363
paper_authors.name: Samuel L. Smith
paper_authors._id: 65e155a99944a44329e45364
paper_authors.name: Anushan Fernando
paper_authors._id: 65e155a99944a44329e45365
paper_authors.name: Aleksandar Botev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:54:47.412Z
paper_authors._id: 65d6647110d158dece540b5c
paper_authors.avatarUrl: /avatars/fc0bbdeb0c23ceb9afbf2a658fc888f2.svg
paper_authors.fullname: Aleksandar Botev
paper_authors.type: user
paper_authors.user: botevmg
paper_authors._id: 65e155a99944a44329e45366
paper_authors.name: George Cristian-Muraru
paper_authors._id: 65e155a99944a44329e45367
paper_authors.name: Albert Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:55:57.972Z
paper_authors._id: 642619b62864ea9e59de834e
paper_authors.avatarUrl: /avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
paper_authors.fullname: Albert Gu
paper_authors.type: user
paper_authors.user: albertgu
paper_authors._id: 65e155a99944a44329e45368
paper_authors.name: Ruba Haroun
paper_authors._id: 65e155a99944a44329e45369
paper_authors.name: Leonard Berrada
paper_authors._id: 65e155a99944a44329e4536a
paper_authors.name: Yutian Chen
paper_authors._id: 65e155a99944a44329e4536b
paper_authors.name: Srivatsan Srinivasan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:56:33.595Z
paper_authors._id: 64f5121aa30d6b0a26c1b7e1
paper_authors.avatarUrl: /avatars/862fe8ed200e4759835661bab161ad09.svg
paper_authors.fullname: Srivatsan Srinivasan
paper_authors.type: user
paper_authors.user: srivatr
paper_authors._id: 65e155a99944a44329e4536c
paper_authors.name: Guillaume Desjardins
paper_authors._id: 65e155a99944a44329e4536d
paper_authors.name: Arnaud Doucet
paper_authors._id: 65e155a99944a44329e4536e
paper_authors.name: David Budden
paper_authors._id: 65e155a99944a44329e4536f
paper_authors.name: Yee Whye Teh
paper_authors._id: 65e155a99944a44329e45370
paper_authors.name: Razvan Pascanu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:56:57.997Z
paper_authors._id: 64b9310403124195cd9778ec
paper_authors.avatarUrl: /avatars/57c594d3d0f97d3010b15b6a0806451c.svg
paper_authors.fullname: Razvan Pascanu
paper_authors.type: user
paper_authors.user: razp
paper_authors._id: 65e155a99944a44329e45371
paper_authors.name: Nando De Freitas
paper_authors._id: 65e155a99944a44329e45372
paper_authors.name: Caglar Gulcehre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:57:08.266Z
paper_authors._id: 64d28085db0b63b977a6c3f8
paper_authors.avatarUrl: /avatars/994cc3ff7c18650fe2535e473d56b240.svg
paper_authors.fullname: Caglar Gulcehre
paper_authors.type: user
paper_authors.user: caglarg
paper_title: Griffin: Mixing Gated Linear Recurrences with Local Attention for
    Efficient Language Models
paper_summary: Recurrent neural networks (RNNs) have fast inference and scale efficiently on
  long sequences, but they are difficult to train and hard to scale. We propose
  Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that
  mixes gated linear recurrences with local attention. Hawk exceeds the reported
  performance of Mamba on downstream tasks, while Griffin matches the performance
  of Llama-2 despite being trained on over 6 times fewer tokens. We also show
  that Griffin can extrapolate on sequences significantly longer than those seen
  during training. Our models match the hardware efficiency of Transformers
  during training, and during inference they have lower latency and significantly
  higher throughput. We scale Griffin up to 14B parameters, and explain how to
  shard our models for efficient distributed training.
paper_ai_summary: Hawk and Griffin, models incorporating gated linear recurrences, achieve high performance with lower computational resources and better hardware efficiency compared to existing architectures.
paper_ai_keywords: RNNs gated linear recurrences local attention Mamba Llama-2 tokens extrapolation hardware efficiency Transformers sharding