title: RecurrentGemma: Moving Past Transformers for Efficient Open Language
    Models
summary: We introduce RecurrentGemma, an open language model which uses Google's novel
  Griffin architecture. Griffin combines linear recurrences with local attention
  to achieve excellent performance on language. It has a fixed-sized state, which
  reduces memory use and enables efficient inference on long sequences. We
  provide a pre-trained model with 2B non-embedding parameters, and an
  instruction tuned variant. Both models achieve comparable performance to
  Gemma-2B despite being trained on fewer tokens.
paper_authors._id: 661890a365e0419878fcb603
paper_authors.name: Aleksandar Botev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:01:11.116Z
paper_authors._id: 65d6647110d158dece540b5c
paper_authors.avatarUrl: /avatars/fc0bbdeb0c23ceb9afbf2a658fc888f2.svg
paper_authors.fullname: Aleksandar Botev
paper_authors.type: user
paper_authors.user: botevmg
paper_authors._id: 661890a365e0419878fcb604
paper_authors.name: Soham De
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:01:25.519Z
paper_authors._id: 661415d9f9e58fe02bd8661d
paper_authors.avatarUrl: /avatars/7a28bb507b8bac72a8b161e774b19fbd.svg
paper_authors.fullname: Soham De
paper_authors.type: user
paper_authors.user: sohamde
paper_authors._id: 661890a365e0419878fcb605
paper_authors.name: Samuel L Smith
paper_authors._id: 661890a365e0419878fcb606
paper_authors.name: Anushan Fernando
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:01:44.512Z
paper_authors._id: 660c2ccb3aa98dc4b27de7b3
paper_authors.avatarUrl: /avatars/2b19094e1b876a1be1f763dc941ff0bf.svg
paper_authors.fullname: Anushan Fernando
paper_authors.type: user
paper_authors.user: AnushanF
paper_authors._id: 661890a365e0419878fcb607
paper_authors.name: George-Cristian Muraru
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:01:50.783Z
paper_authors._id: 660dd120fc9742e05187cffd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/660dd120fc9742e05187cffd/bFZHJMJcN6rJGo_Yala89.png
paper_authors.fullname: George-Cristian Muraru
paper_authors.type: user
paper_authors.user: gmuraru
paper_authors._id: 661890a365e0419878fcb608
paper_authors.name: Ruba Haroun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:01:57.539Z
paper_authors._id: 66167eec24c94fd8a2427513
paper_authors.avatarUrl: /avatars/24ff7acfbf064a3a37dcad25ef9d45ee.svg
paper_authors.fullname: Ruba Haroun
paper_authors.type: user
paper_authors.user: RubaHaroun
paper_authors._id: 661890a365e0419878fcb609
paper_authors.name: Leonard Berrada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:02:04.421Z
paper_authors._id: 65fc77aad6206abe6cb611f1
paper_authors.avatarUrl: /avatars/306334124d90d1b4d4f82fb47f56156f.svg
paper_authors.fullname: Leonard Berrada
paper_authors.type: user
paper_authors.user: lberrada
paper_authors._id: 661890a365e0419878fcb60a
paper_authors.name: Razvan Pascanu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:02:10.349Z
paper_authors._id: 64b9310403124195cd9778ec
paper_authors.avatarUrl: /avatars/57c594d3d0f97d3010b15b6a0806451c.svg
paper_authors.fullname: Razvan Pascanu
paper_authors.type: user
paper_authors.user: razp
paper_authors._id: 661890a365e0419878fcb60b
paper_authors.name: Pier Giuseppe Sessa
paper_authors._id: 661890a365e0419878fcb60c
paper_authors.name: Robert Dadashi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:02:23.830Z
paper_authors._id: 65fc7c0da57b88a765aea493
paper_authors.avatarUrl: /avatars/440f88769da1113d6158fe7e0514ead3.svg
paper_authors.fullname: Robert Dadashi
paper_authors.type: user
paper_authors.user: ddsh
paper_authors._id: 661890a365e0419878fcb60d
paper_authors.name: Léonard Hussenot
paper_authors._id: 661890a365e0419878fcb60e
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:02:35.672Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 661890a365e0419878fcb60f
paper_authors.name: Sertan Girgin
paper_authors._id: 661890a365e0419878fcb610
paper_authors.name: Olivier Bachem
paper_authors._id: 661890a365e0419878fcb611
paper_authors.name: Alek Andreev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:02:54.909Z
paper_authors._id: 65d667858ac5cf899e3b66c4
paper_authors.avatarUrl: /avatars/340468ee8cd9125eceae23a9b8e536a5.svg
paper_authors.fullname: Alek Andreev
paper_authors.type: user
paper_authors.user: alekandreev
paper_authors._id: 661890a365e0419878fcb612
paper_authors.name: Kathleen Kenealy
paper_authors._id: 661890a365e0419878fcb613
paper_authors.name: Thomas Mesnard
paper_authors._id: 661890a365e0419878fcb614
paper_authors.name: Cassidy Hardin
paper_authors._id: 661890a365e0419878fcb615
paper_authors.name: Surya Bhupatiraju
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:03:24.765Z
paper_authors._id: 65d7644d19bf1e60614dee6e
paper_authors.avatarUrl: /avatars/422a8d28fa9d9ff2c3ec1406352cb346.svg
paper_authors.fullname: Surya Bhupatiraju
paper_authors.type: user
paper_authors.user: suryabhupa
paper_authors._id: 661890a365e0419878fcb616
paper_authors.name: Shreya Pathak
paper_authors._id: 661890a365e0419878fcb617
paper_authors.name: Laurent Sifre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:03:35.899Z
paper_authors._id: 65d74b9b7713dc8fd8956d1c
paper_authors.avatarUrl: /avatars/5d5873861134bb079753c48304a7b7c2.svg
paper_authors.fullname: Laurent Sifre
paper_authors.type: user
paper_authors.user: laurentsifre
paper_authors._id: 661890a365e0419878fcb618
paper_authors.name: Morgane Rivière
paper_authors._id: 661890a365e0419878fcb619
paper_authors.name: Mihir Sanjay Kale
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:03:50.288Z
paper_authors._id: 6226953f7551f131b4eba5c5
paper_authors.avatarUrl: /avatars/eec8fa43fc37f546d9fef118a138d8a4.svg
paper_authors.fullname: Mihir Sanjay Kale
paper_authors.type: user
paper_authors.user: mihirkale
paper_authors._id: 661890a365e0419878fcb61a
paper_authors.name: Juliette Love
paper_authors._id: 661890a365e0419878fcb61b
paper_authors.name: Pouya Tafti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:04:09.855Z
paper_authors._id: 6558091ecafc48de36f848a8
paper_authors.avatarUrl: /avatars/3b7e8327d369436d02fe8c58dde036c2.svg
paper_authors.fullname: Pouya Tafti
paper_authors.type: user
paper_authors.user: pdt
paper_authors._id: 661890a365e0419878fcb61c
paper_authors.name: Armand Joulin
paper_authors._id: 661890a365e0419878fcb61d
paper_authors.name: Noah Fiedel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:04:21.963Z
paper_authors._id: 62f82ebd870a3f98bbf9e62a
paper_authors.avatarUrl: /avatars/3b03299e128abf2c8ba82db704855de7.svg
paper_authors.fullname: Noah Fiedel
paper_authors.type: user
paper_authors.user: nfiedel
paper_authors._id: 661890a365e0419878fcb61e
paper_authors.name: Evan Senter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:04:30.429Z
paper_authors._id: 62a47ec0fdd7a13aac264904
paper_authors.avatarUrl: /avatars/f2ff23645203ed2399cfe8b447af437a.svg
paper_authors.fullname: Evan Senter
paper_authors.type: user
paper_authors.user: evansenter
paper_authors._id: 661890a365e0419878fcb61f
paper_authors.name: Yutian Chen
paper_authors._id: 661890a365e0419878fcb620
paper_authors.name: Srivatsan Srinivasan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:04:56.234Z
paper_authors._id: 64f5121aa30d6b0a26c1b7e1
paper_authors.avatarUrl: /avatars/862fe8ed200e4759835661bab161ad09.svg
paper_authors.fullname: Srivatsan Srinivasan
paper_authors.type: user
paper_authors.user: srivatr
paper_authors._id: 661890a365e0419878fcb621
paper_authors.name: Guillaume Desjardins
paper_authors._id: 661890a365e0419878fcb622
paper_authors.name: David Budden
paper_authors._id: 661890a365e0419878fcb623
paper_authors.name: Arnaud Doucet
paper_authors._id: 661890a365e0419878fcb624
paper_authors.name: Sharad Vikram
paper_authors._id: 661890a365e0419878fcb625
paper_authors.name: Adam Paszke
paper_authors._id: 661890a365e0419878fcb626
paper_authors.name: Trevor Gale
paper_authors._id: 661890a365e0419878fcb627
paper_authors.name: Sebastian Borgeaud
paper_authors._id: 661890a365e0419878fcb628
paper_authors.name: Charlie Chen
paper_authors._id: 661890a365e0419878fcb629
paper_authors.name: Andy Brock
paper_authors._id: 661890a365e0419878fcb62a
paper_authors.name: Antonia Paterson
paper_authors._id: 661890a365e0419878fcb62b
paper_authors.name: Jenny Brennan
paper_authors._id: 661890a365e0419878fcb62c
paper_authors.name: Meg Risdal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-12T08:28:39.010Z
paper_authors._id: 626b23fc822e3d85324babed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
paper_authors.fullname: Meg Risdal
paper_authors.type: user
paper_authors.user: megrisdal
paper_authors._id: 661890a365e0419878fcb62d
paper_authors.name: Raj Gundluru
paper_authors._id: 661890a365e0419878fcb62e
paper_authors.name: Nesh Devanathan
paper_authors._id: 661890a365e0419878fcb62f
paper_authors.name: Paul Mooney
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:05:57.471Z
paper_authors._id: 644fadb9fc5e5ff919aed863
paper_authors.avatarUrl: /avatars/9b646ff3eeda1f01cc2b1cb86d1634f9.svg
paper_authors.fullname: Paul Mooney
paper_authors.type: user
paper_authors.user: paulmooney
paper_authors._id: 661890a365e0419878fcb630
paper_authors.name: Nilay Chauhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:07:38.639Z
paper_authors._id: 641415cd5189abde8cbbd900
paper_authors.avatarUrl: /avatars/401807f3174492ab64b0197427f21e0b.svg
paper_authors.fullname: Nilay Chauhan
paper_authors.type: user
paper_authors.user: nilaychauhan
paper_authors._id: 661890a365e0419878fcb631
paper_authors.name: Phil Culliton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:07:30.197Z
paper_authors._id: 627a9aa0f967ef475abe8037
paper_authors.avatarUrl: /avatars/47cf94d6c015577d0a860b617c9ea2b8.svg
paper_authors.fullname: Phil Culliton
paper_authors.type: user
paper_authors.user: PhilCulliton
paper_authors._id: 661890a365e0419878fcb632
paper_authors.name: Luiz GUStavo Martins
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:07:22.567Z
paper_authors._id: 637a4a31e20e8765c5220b77
paper_authors.avatarUrl: /avatars/eb8e5b3bf2c1c99dad4f35ac24788669.svg
paper_authors.fullname: Luiz Gustavo Martins
paper_authors.type: user
paper_authors.user: gusthema
paper_authors._id: 661890a365e0419878fcb633
paper_authors.name: Elisa Bandy
paper_authors._id: 661890a365e0419878fcb634
paper_authors.name: David Huntsperger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:07:10.781Z
paper_authors._id: 65f35e8bc2d214f884656a67
paper_authors.avatarUrl: /avatars/2a46a832328a1269d0077097f6a121ca.svg
paper_authors.fullname: David Huntsperger
paper_authors.type: user
paper_authors.user: dhuntsperger
paper_authors._id: 661890a365e0419878fcb635
paper_authors.name: Glenn Cameron
paper_authors._id: 661890a365e0419878fcb636
paper_authors.name: Arthur Zucker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T14:49:59.500Z
paper_authors._id: 62441cb7456803e95009a08f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg
paper_authors.fullname: Arthur Zucker
paper_authors.type: user
paper_authors.user: ArthurZ
paper_authors._id: 661890a365e0419878fcb637
paper_authors.name: Tris Warkentin
paper_authors._id: 661890a365e0419878fcb638
paper_authors.name: Ludovic Peran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:06:46.212Z
paper_authors._id: 60e473dcdefbf317ecd88035
paper_authors.avatarUrl: /avatars/e5c069e6cfc73ea5d3d6c2a437490cdd.svg
paper_authors.fullname: LUDOVIC PERAN
paper_authors.type: user
paper_authors.user: LudovicPeran
paper_authors._id: 661890a365e0419878fcb639
paper_authors.name: Minh Giang
paper_authors._id: 661890a365e0419878fcb63a
paper_authors.name: Zoubin Ghahramani
paper_authors._id: 661890a365e0419878fcb63b
paper_authors.name: Clément Farabet
paper_authors._id: 661890a365e0419878fcb63c
paper_authors.name: Koray Kavukcuoglu
paper_authors._id: 661890a365e0419878fcb63d
paper_authors.name: Demis Hassabis
paper_authors._id: 661890a365e0419878fcb63e
paper_authors.name: Raia Hadsell
paper_authors._id: 661890a365e0419878fcb63f
paper_authors.name: Yee Whye Teh
paper_authors._id: 661890a365e0419878fcb640
paper_authors.name: Nando de Frietas
paper_title: RecurrentGemma: Moving Past Transformers for Efficient Open Language
    Models
paper_summary: We introduce RecurrentGemma, an open language model which uses Google's novel
  Griffin architecture. Griffin combines linear recurrences with local attention
  to achieve excellent performance on language. It has a fixed-sized state, which
  reduces memory use and enables efficient inference on long sequences. We
  provide a pre-trained model with 2B non-embedding parameters, and an
  instruction tuned variant. Both models achieve comparable performance to
  Gemma-2B despite being trained on fewer tokens.
paper_ai_keywords: RecurrentGemma Griffin architecture linear recurrences local attention fixed-sized state memory use efficient inference long sequences pre-trained model non-embedding parameters instruction tuned variant Gemma-2B