title: Gemma 2: Improving Open Language Models at a Practical Size
summary: In this work, we introduce Gemma 2, a new addition to the Gemma family of
  lightweight, state-of-the-art open models, ranging in scale from 2 billion to
  27 billion parameters. In this new version, we apply several known technical
  modifications to the Transformer architecture, such as interleaving
  local-global attentions (Beltagy et al., 2020a) and group-query attention
  (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge
  distillation (Hinton et al., 2015) instead of next token prediction. The
  resulting models deliver the best performance for their size, and even offer
  competitive alternatives to models that are 2-3 times bigger. We release all
  our models to the community.
paper_authors._id: 66aca7a3eb69d5b0eba59951
paper_authors.name: Gemma Team
paper_authors._id: 66aca7a3eb69d5b0eba59952
paper_authors.name: Morgane Riviere
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:21:44.445Z
paper_authors._id: 66953d345d5615345518ca31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66953d345d5615345518ca31/rIpTxlrMxVNB9Td9fJijl.png
paper_authors.fullname: Morgane Riviere
paper_authors.type: user
paper_authors.user: morugan
paper_authors._id: 66aca7a3eb69d5b0eba59953
paper_authors.name: Shreya Pathak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:21:51.931Z
paper_authors._id: 667125edfc615a6257be29bc
paper_authors.avatarUrl: /avatars/c1b191d6443b7a4ac52866a746ca5717.svg
paper_authors.fullname: Shreya Pathak
paper_authors.type: user
paper_authors.user: shreyapathak
paper_authors._id: 66aca7a3eb69d5b0eba59954
paper_authors.name: Pier Giuseppe Sessa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:01.722Z
paper_authors._id: 667a7876becec8fc5112cf1f
paper_authors.avatarUrl: /avatars/6874f0e5a7edca4145c57d7908f89840.svg
paper_authors.fullname: Pier Giuseppe Sessa
paper_authors.type: user
paper_authors.user: piergs
paper_authors._id: 66aca7a3eb69d5b0eba59955
paper_authors.name: Cassidy Hardin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:07.910Z
paper_authors._id: 663a5ec5a4152b77b6cf96fd
paper_authors.avatarUrl: /avatars/a51901b21d8670e26b2ac57fd482bb63.svg
paper_authors.fullname: Cassidy Hardin
paper_authors.type: user
paper_authors.user: casshardin
paper_authors._id: 66aca7a3eb69d5b0eba59956
paper_authors.name: Surya Bhupatiraju
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:14.718Z
paper_authors._id: 65d7644d19bf1e60614dee6e
paper_authors.avatarUrl: /avatars/422a8d28fa9d9ff2c3ec1406352cb346.svg
paper_authors.fullname: Surya Bhupatiraju
paper_authors.type: user
paper_authors.user: suryabhupa
paper_authors._id: 66aca7a3eb69d5b0eba59957
paper_authors.name: Léonard Hussenot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:24.134Z
paper_authors._id: 667a7b2e1764931987803b46
paper_authors.avatarUrl: /avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
paper_authors.fullname: Léonard Hussenot
paper_authors.type: user
paper_authors.user: leonardhussenot
paper_authors._id: 66aca7a3eb69d5b0eba59958
paper_authors.name: Thomas Mesnard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:33.910Z
paper_authors._id: 6639adab8eb8c9c62ac3fb15
paper_authors.avatarUrl: /avatars/83133b04922143be1d8a1fb71fb2938f.svg
paper_authors.fullname: Thomas Mesnard
paper_authors.type: user
paper_authors.user: thomasmesnard
paper_authors._id: 66aca7a3eb69d5b0eba59959
paper_authors.name: Bobak Shahriari
paper_authors._id: 66aca7a3eb69d5b0eba5995a
paper_authors.name: Alexandre Ramé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:46.308Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 66aca7a3eb69d5b0eba5995b
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:22:53.748Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 66aca7a3eb69d5b0eba5995c
paper_authors.name: Peter Liu
paper_authors._id: 66aca7a3eb69d5b0eba5995d
paper_authors.name: Pouya Tafti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:36:32.170Z
paper_authors._id: 6558091ecafc48de36f848a8
paper_authors.avatarUrl: /avatars/3b7e8327d369436d02fe8c58dde036c2.svg
paper_authors.fullname: Pouya Tafti
paper_authors.type: user
paper_authors.user: pdt
paper_authors._id: 66aca7a3eb69d5b0eba5995e
paper_authors.name: Abe Friesen
paper_authors._id: 66aca7a3eb69d5b0eba5995f
paper_authors.name: Michelle Casbon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:36:42.087Z
paper_authors._id: 662806c1167ee93ec8e503a4
paper_authors.avatarUrl: /avatars/8dcf47f91328a38833636068686a8a5a.svg
paper_authors.fullname: Michelle Casbon
paper_authors.type: user
paper_authors.user: michellecasbon
paper_authors._id: 66aca7a3eb69d5b0eba59960
paper_authors.name: Sabela Ramos
paper_authors._id: 66aca7a3eb69d5b0eba59961
paper_authors.name: Ravin Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:36:52.026Z
paper_authors._id: 6317476e106740aaad50e2cd
paper_authors.avatarUrl: /avatars/7f066cb7b99f24b57f87598bc9ba7417.svg
paper_authors.fullname: Ravin Kumar
paper_authors.type: user
paper_authors.user: canyon289
paper_authors._id: 66aca7a3eb69d5b0eba59962
paper_authors.name: Charline Le Lan
paper_authors._id: 66aca7a3eb69d5b0eba59963
paper_authors.name: Sammy Jerome
paper_authors._id: 66aca7a3eb69d5b0eba59964
paper_authors.name: Anton Tsitsulin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:37:07.767Z
paper_authors._id: 63ab8e5c8a357d5057ae659e
paper_authors.avatarUrl: /avatars/0eed94fcdb6d7c86259bc5b72b5bd41e.svg
paper_authors.fullname: Anton Tsitsulin
paper_authors.type: user
paper_authors.user: xgfs
paper_authors._id: 66aca7a3eb69d5b0eba59965
paper_authors.name: Nino Vieillard
paper_authors._id: 66aca7a3eb69d5b0eba59966
paper_authors.name: Piotr Stanczyk
paper_authors._id: 66aca7a3eb69d5b0eba59967
paper_authors.name: Sertan Girgin
paper_authors._id: 66aca7a3eb69d5b0eba59968
paper_authors.name: Nikola Momchev
paper_authors._id: 66aca7a3eb69d5b0eba59969
paper_authors.name: Matt Hoffman
paper_authors._id: 66aca7a3eb69d5b0eba5996a
paper_authors.name: Shantanu Thakoor
paper_authors._id: 66aca7a3eb69d5b0eba5996b
paper_authors.name: Jean-Bastien Grill
paper_authors._id: 66aca7a3eb69d5b0eba5996c
paper_authors.name: Behnam Neyshabur
paper_authors._id: 66aca7a3eb69d5b0eba5996d
paper_authors.name: Alanna Walton
paper_authors._id: 66aca7a3eb69d5b0eba5996e
paper_authors.name: Aliaksei Severyn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:38:40.223Z
paper_authors._id: 66ac94b19c108b7a2af462c5
paper_authors.avatarUrl: /avatars/dc1b38b62dc58b8a0f19fd4978f7bb8c.svg
paper_authors.fullname: Aliaksei Severyn
paper_authors.type: user
paper_authors.user: aseveryn
paper_authors._id: 66aca7a3eb69d5b0eba5996f
paper_authors.name: Alicia Parrish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:38:46.584Z
paper_authors._id: 659d71eaf0ae1a57ae140831
paper_authors.avatarUrl: /avatars/b9eb8c70b4fe5dbd0f1555602ae41b65.svg
paper_authors.fullname: Alicia Parrish
paper_authors.type: user
paper_authors.user: avparrish
paper_authors._id: 66aca7a3eb69d5b0eba59970
paper_authors.name: Aliya Ahmad
paper_authors._id: 66aca7a3eb69d5b0eba59971
paper_authors.name: Allen Hutchison
paper_authors._id: 66aca7a3eb69d5b0eba59972
paper_authors.name: Alvin Abdagic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:39:19.565Z
paper_authors._id: 66aaa23c08609915b3855f0b
paper_authors.avatarUrl: /avatars/6e6c105d921c41477b61af7a4a1b6495.svg
paper_authors.fullname: Alvin Abdagic
paper_authors.type: user
paper_authors.user: aabdagic
paper_authors._id: 66aca7a3eb69d5b0eba59973
paper_authors.name: Amanda Carl
paper_authors._id: 66aca7a3eb69d5b0eba59974
paper_authors.name: Amy Shen
paper_authors._id: 66aca7a3eb69d5b0eba59975
paper_authors.name: Andy Brock
paper_authors._id: 66aca7a3eb69d5b0eba59976
paper_authors.name: Andy Coenen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:40:04.549Z
paper_authors._id: 63b334ff922f26a27e79cf1e
paper_authors.avatarUrl: /avatars/a3110658bae1a7c9591c096c6ac457f6.svg
paper_authors.fullname: Andy Coenen
paper_authors.type: user
paper_authors.user: cannoneyed
paper_authors._id: 66aca7a3eb69d5b0eba59977
paper_authors.name: Anthony Laforge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:40:12.260Z
paper_authors._id: 642cd49253e76b4c22872896
paper_authors.avatarUrl: /avatars/ff2bab2077064d16082f0a7592e821ba.svg
paper_authors.fullname: Anthony Laforge
paper_authors.type: user
paper_authors.user: aglaforge
paper_authors._id: 66aca7a3eb69d5b0eba59978
paper_authors.name: Antonia Paterson
paper_authors._id: 66aca7a3eb69d5b0eba59979
paper_authors.name: Ben Bastian
paper_authors._id: 66aca7a3eb69d5b0eba5997a
paper_authors.name: Bilal Piot
paper_authors._id: 66aca7a3eb69d5b0eba5997b
paper_authors.name: Bo Wu
paper_authors._id: 66aca7a3eb69d5b0eba5997c
paper_authors.name: Brandon Royal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:40:39.045Z
paper_authors._id: 64e4f2bfccb9ee90b10ffdba
paper_authors.avatarUrl: /avatars/49a28f87b7280c47d2b10cf0fd35d332.svg
paper_authors.fullname: Brandon Royal
paper_authors.type: user
paper_authors.user: broyal
paper_authors._id: 66aca7a3eb69d5b0eba5997d
paper_authors.name: Charlie Chen
paper_authors._id: 66aca7a3eb69d5b0eba5997e
paper_authors.name: Chintu Kumar
paper_authors._id: 66aca7a3eb69d5b0eba5997f
paper_authors.name: Chris Perry
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:41:43.466Z
paper_authors._id: 630cf8e33dc31beba6eeadfc
paper_authors.avatarUrl: /avatars/f66eaecb7e710e58f0c06687503c106d.svg
paper_authors.fullname: Chris Perry
paper_authors.type: user
paper_authors.user: cperry
paper_authors._id: 66aca7a3eb69d5b0eba59980
paper_authors.name: Chris Welty
paper_authors._id: 66aca7a3eb69d5b0eba59981
paper_authors.name: Christopher A. Choquette-Choo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:41:55.951Z
paper_authors._id: 649c5307e89a7650158f4965
paper_authors.avatarUrl: /avatars/3c71b501db27c86ed686ae26ad1391cd.svg
paper_authors.fullname: Christopher A. Choquette-Choo
paper_authors.type: user
paper_authors.user: cchoquette
paper_authors._id: 66aca7a3eb69d5b0eba59982
paper_authors.name: Danila Sinopalnikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:42:04.581Z
paper_authors._id: 667a6ddc400b4b8da62488f1
paper_authors.avatarUrl: /avatars/65772bd92bbbe832eeb51298a231f57a.svg
paper_authors.fullname: Danila Sinopalnikov
paper_authors.type: user
paper_authors.user: sinopalnikov
paper_authors._id: 66aca7a3eb69d5b0eba59983
paper_authors.name: David Weinberger
paper_authors._id: 66aca7a3eb69d5b0eba59984
paper_authors.name: Dimple Vijaykumar
paper_authors._id: 66aca7a3eb69d5b0eba59985
paper_authors.name: Dominika Rogozińska
paper_authors._id: 66aca7a3eb69d5b0eba59986
paper_authors.name: Dustin Herbison
paper_authors._id: 66aca7a3eb69d5b0eba59987
paper_authors.name: Elisa Bandy
paper_authors._id: 66aca7a3eb69d5b0eba59988
paper_authors.name: Emma Wang
paper_authors._id: 66aca7a3eb69d5b0eba59989
paper_authors.name: Eric Noland
paper_authors._id: 66aca7a3eb69d5b0eba5998a
paper_authors.name: Erica Moreira
paper_authors._id: 66aca7a3eb69d5b0eba5998b
paper_authors.name: Evan Senter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:43:51.964Z
paper_authors._id: 62a47ec0fdd7a13aac264904
paper_authors.avatarUrl: /avatars/f2ff23645203ed2399cfe8b447af437a.svg
paper_authors.fullname: Evan Senter
paper_authors.type: user
paper_authors.user: evansenter
paper_authors._id: 66aca7a3eb69d5b0eba5998c
paper_authors.name: Evgenii Eltyshev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:43:59.687Z
paper_authors._id: 638b4f0a3bbf29e589029a2e
paper_authors.avatarUrl: /avatars/d39fcf06b93affb6d07f534b952c6c16.svg
paper_authors.fullname: Evgenii Eltyshev
paper_authors.type: user
paper_authors.user: EvgeniiEltyshev
paper_authors._id: 66aca7a3eb69d5b0eba5998d
paper_authors.name: Francesco Visin
paper_authors._id: 66aca7a3eb69d5b0eba5998e
paper_authors.name: Gabriel Rasskin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:44:10.319Z
paper_authors._id: 63174c76c92fd6fee319e7f5
paper_authors.avatarUrl: /avatars/a2a71b9eed5aa97925740290b2c41dc6.svg
paper_authors.fullname: Gabriel Rasskin
paper_authors.type: user
paper_authors.user: grasskin
paper_authors._id: 66aca7a3eb69d5b0eba5998f
paper_authors.name: Gary Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:44:43.648Z
paper_authors._id: 66a138cc627519df4aa4b7ab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66a138cc627519df4aa4b7ab/XdIE8H9v22kvsWweMDF6d.jpeg
paper_authors.fullname: Gary Wei
paper_authors.type: user
paper_authors.user: cdwei-gg
paper_authors._id: 66aca7a3eb69d5b0eba59990
paper_authors.name: Glenn Cameron
paper_authors._id: 66aca7a3eb69d5b0eba59991
paper_authors.name: Gus Martins
paper_authors._id: 66aca7a3eb69d5b0eba59992
paper_authors.name: Hadi Hashemi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:45:20.378Z
paper_authors._id: 64da4f7578864cd2221af0c1
paper_authors.avatarUrl: /avatars/428d3fd717654893984f9d01f3d6667d.svg
paper_authors.fullname: Hadi Hashemi
paper_authors.type: user
paper_authors.user: hadi-hashemi
paper_authors._id: 66aca7a3eb69d5b0eba59993
paper_authors.name: Hanna Klimczak-Plucińska
paper_authors._id: 66aca7a3eb69d5b0eba59994
paper_authors.name: Harleen Batra
paper_authors._id: 66aca7a3eb69d5b0eba59995
paper_authors.name: Harsh Dhand
paper_authors._id: 66aca7a3eb69d5b0eba59996
paper_authors.name: Ivan Nardini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:45:56.728Z
paper_authors._id: 6280f4b052815a0dc7399902
paper_authors.avatarUrl: /avatars/e098c20e16d17e687d5d0c78745cdc7d.svg
paper_authors.fullname: Ivan Nardini
paper_authors.type: user
paper_authors.user: IlNardo92
paper_authors._id: 66aca7a3eb69d5b0eba59997
paper_authors.name: Jacinda Mein
paper_authors._id: 66aca7a3eb69d5b0eba59998
paper_authors.name: Jack Zhou
paper_authors._id: 66aca7a3eb69d5b0eba59999
paper_authors.name: James Svensson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:46:11.238Z
paper_authors._id: 62d33483c20456076a2a2c06
paper_authors.avatarUrl: /avatars/e1ac677101381030a3174cd859604afa.svg
paper_authors.fullname: James Svensson
paper_authors.type: user
paper_authors.user: Sven84
paper_authors._id: 66aca7a3eb69d5b0eba5999a
paper_authors.name: Jeff Stanway
paper_authors._id: 66aca7a3eb69d5b0eba5999b
paper_authors.name: Jetha Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:46:31.465Z
paper_authors._id: 6670e8966d34fe6262885205
paper_authors.avatarUrl: /avatars/f5f7845a54c2bf8e418989f113272578.svg
paper_authors.fullname: Jetha Chan
paper_authors.type: user
paper_authors.user: jethac
paper_authors._id: 66aca7a3eb69d5b0eba5999c
paper_authors.name: Jin Zhou
paper_authors._id: 66aca7a3eb69d5b0eba5999d
paper_authors.name: Joana Carrasqueira
paper_authors._id: 66aca7a3eb69d5b0eba5999e
paper_authors.name: Joana Iljazi
paper_authors._id: 66aca7a3eb69d5b0eba5999f
paper_authors.name: Jocelyn Becker
paper_authors._id: 66aca7a3eb69d5b0eba599a0
paper_authors.name: Joe Fernandez
paper_authors._id: 66aca7a3eb69d5b0eba599a1
paper_authors.name: Joost van Amersfoort
paper_authors._id: 66aca7a3eb69d5b0eba599a2
paper_authors.name: Josh Gordon
paper_authors._id: 66aca7a3eb69d5b0eba599a3
paper_authors.name: Josh Lipschultz
paper_authors._id: 66aca7a3eb69d5b0eba599a4
paper_authors.name: Josh Newlan
paper_authors._id: 66aca7a3eb69d5b0eba599a5
paper_authors.name: Ju-yeong Ji
paper_authors._id: 66aca7a3eb69d5b0eba599a6
paper_authors.name: Kareem Mohamed
paper_authors._id: 66aca7a3eb69d5b0eba599a7
paper_authors.name: Kartikeya Badola
paper_authors._id: 66aca7a3eb69d5b0eba599a8
paper_authors.name: Kat Black
paper_authors._id: 66aca7a3eb69d5b0eba599a9
paper_authors.name: Katie Millican
paper_authors._id: 66aca7a3eb69d5b0eba599aa
paper_authors.name: Keelin McDonell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:56:57.135Z
paper_authors._id: 62b9e7c04dde8074b78be449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675986645321-62b9e7c04dde8074b78be449.jpeg
paper_authors.fullname: Keelin McDonell
paper_authors.type: user
paper_authors.user: keelin
paper_authors._id: 66aca7a3eb69d5b0eba599ab
paper_authors.name: Kelvin Nguyen
paper_authors._id: 66aca7a3eb69d5b0eba599ac
paper_authors.name: Kiranbir Sodhia
paper_authors._id: 66aca7a3eb69d5b0eba599ad
paper_authors.name: Kish Greene
paper_authors._id: 66aca7a3eb69d5b0eba599ae
paper_authors.name: Lars Lowe Sjoesund
paper_authors._id: 66aca7a3eb69d5b0eba599af
paper_authors.name: Lauren Usui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:00:24.875Z
paper_authors._id: 63dd3ddea8877129a15a534c
paper_authors.avatarUrl: /avatars/96bdadf176162856d1687f8298915484.svg
paper_authors.fullname: lauren usui
paper_authors.type: user
paper_authors.user: lusui
paper_authors._id: 66aca7a3eb69d5b0eba599b0
paper_authors.name: Laurent Sifre
paper_authors._id: 66aca7a3eb69d5b0eba599b1
paper_authors.name: Lena Heuermann
paper_authors._id: 66aca7a3eb69d5b0eba599b2
paper_authors.name: Leticia Lago
paper_authors._id: 66aca7a3eb69d5b0eba599b3
paper_authors.name: Lilly McNealus
paper_authors._id: 66aca7a3eb69d5b0eba599b4
paper_authors.name: Livio Baldini Soares
paper_authors._id: 66aca7a3eb69d5b0eba599b5
paper_authors.name: Logan Kilpatrick
paper_authors._id: 66aca7a3eb69d5b0eba599b6
paper_authors.name: Lucas Dixon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:57:55.078Z
paper_authors._id: 64c25a9c2b1ffc2eecbce31d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c25a9c2b1ffc2eecbce31d/uTp3yOPdmCSAmhzWwKUjK.png
paper_authors.fullname: Lucas Dixon
paper_authors.type: user
paper_authors.user: iislucas
paper_authors._id: 66aca7a3eb69d5b0eba599b7
paper_authors.name: Luciano Martins
paper_authors._id: 66aca7a3eb69d5b0eba599b8
paper_authors.name: Machel Reid
paper_authors._id: 66aca7a3eb69d5b0eba599b9
paper_authors.name: Manvinder Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:57:17.021Z
paper_authors._id: 64dc01ab3764a5137e8fbd70
paper_authors.avatarUrl: /avatars/807bef46ed74bbcc6bd9cac9b9acf82b.svg
paper_authors.fullname: Manvinder Singh
paper_authors.type: user
paper_authors.user: manvinder
paper_authors._id: 66aca7a3eb69d5b0eba599ba
paper_authors.name: Mark Iverson
paper_authors._id: 66aca7a3eb69d5b0eba599bb
paper_authors.name: Martin Görner
paper_authors._id: 66aca7a3eb69d5b0eba599bc
paper_authors.name: Mat Velloso
paper_authors._id: 66aca7a3eb69d5b0eba599bd
paper_authors.name: Mateo Wirth
paper_authors._id: 66aca7a3eb69d5b0eba599be
paper_authors.name: Matt Davidow
paper_authors._id: 66aca7a3eb69d5b0eba599bf
paper_authors.name: Matt Miller
paper_authors._id: 66aca7a3eb69d5b0eba599c0
paper_authors.name: Matthew Rahtz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:00:57.136Z
paper_authors._id: 6691892b389abe9d474d80da
paper_authors.avatarUrl: /avatars/7b2fa384c7f1fcaef1e99d297b9a624b.svg
paper_authors.fullname: Matthew Rahtz
paper_authors.type: user
paper_authors.user: gdm-mrahtz
paper_authors._id: 66aca7a3eb69d5b0eba599c1
paper_authors.name: Matthew Watson
paper_authors._id: 66aca7a3eb69d5b0eba599c2
paper_authors.name: Meg Risdal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:00:40.049Z
paper_authors._id: 626b23fc822e3d85324babed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
paper_authors.fullname: Meg Risdal
paper_authors.type: user
paper_authors.user: megrisdal
paper_authors._id: 66aca7a3eb69d5b0eba599c3
paper_authors.name: Mehran Kazemi
paper_authors._id: 66aca7a3eb69d5b0eba599c4
paper_authors.name: Michael Moynihan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:01:19.758Z
paper_authors._id: 65aeae63563e362f5521e32a
paper_authors.avatarUrl: /avatars/ad3a9f68cae8cbaf639084e31adffc54.svg
paper_authors.fullname: Michael Moynihan
paper_authors.type: user
paper_authors.user: mmoynihan
paper_authors._id: 66aca7a3eb69d5b0eba599c5
paper_authors.name: Ming Zhang
paper_authors._id: 66aca7a3eb69d5b0eba599c6
paper_authors.name: Minsuk Kahng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:23:27.543Z
paper_authors._id: 6301f3fb4ccccaa23d38fb7f
paper_authors.avatarUrl: /avatars/4226111991d1788d3d442ffb2f33ea48.svg
paper_authors.fullname: Minsuk Kahng
paper_authors.type: user
paper_authors.user: minsukkahng
paper_authors._id: 66aca7a3eb69d5b0eba599c7
paper_authors.name: Minwoo Park
paper_authors._id: 66aca7a3eb69d5b0eba599c8
paper_authors.name: Mofi Rahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:01:42.741Z
paper_authors._id: 65d5f883ba0ad765ea02bb9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d5f883ba0ad765ea02bb9c/Z0lPfLx6p4i-NAyCslZkN.png
paper_authors.fullname: mofi rahman
paper_authors.type: user
paper_authors.user: mofirahman
paper_authors._id: 66aca7a3eb69d5b0eba599c9
paper_authors.name: Mohit Khatwani
paper_authors._id: 66aca7a3eb69d5b0eba599ca
paper_authors.name: Natalie Dao
paper_authors._id: 66aca7a3eb69d5b0eba599cb
paper_authors.name: Nenshad Bardoliwalla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:58:17.909Z
paper_authors._id: 65d618051a95fdcf7c59086d
paper_authors.avatarUrl: /avatars/05256144ae27b46b0bd2ff4eb0ea6642.svg
paper_authors.fullname: Nenshad Bardoliwalla
paper_authors.type: user
paper_authors.user: nenshadbardoliwalla
paper_authors._id: 66aca7a3eb69d5b0eba599cc
paper_authors.name: Nesh Devanathan
paper_authors._id: 66aca7a3eb69d5b0eba599cd
paper_authors.name: Neta Dumai
paper_authors._id: 66aca7a3eb69d5b0eba599ce
paper_authors.name: Nilay Chauhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:00:17.063Z
paper_authors._id: 66a7935b7959dfd1e912afca
paper_authors.avatarUrl: /avatars/3d31bfeaf0c008b2c9dae99b8d0298b7.svg
paper_authors.fullname: Nilay Chauhan
paper_authors.type: user
paper_authors.user: nilayc
paper_authors._id: 66aca7a3eb69d5b0eba599cf
paper_authors.name: Oscar Wahltinez
paper_authors._id: 66aca7a3eb69d5b0eba599d0
paper_authors.name: Pankil Botarda
paper_authors._id: 66aca7a3eb69d5b0eba599d1
paper_authors.name: Parker Barnes
paper_authors._id: 66aca7a3eb69d5b0eba599d2
paper_authors.name: Paul Barham
paper_authors._id: 66aca7a3eb69d5b0eba599d3
paper_authors.name: Paul Michel
paper_authors._id: 66aca7a3eb69d5b0eba599d4
paper_authors.name: Pengchong Jin
paper_authors._id: 66aca7a3eb69d5b0eba599d5
paper_authors.name: Petko Georgiev
paper_authors._id: 66aca7a3eb69d5b0eba599d6
paper_authors.name: Phil Culliton
paper_authors._id: 66aca7a3eb69d5b0eba599d7
paper_authors.name: Pradeep Kuppala
paper_authors._id: 66aca7a3eb69d5b0eba599d8
paper_authors.name: Ramona Comanescu
paper_authors._id: 66aca7a3eb69d5b0eba599d9
paper_authors.name: Ramona Merhej
paper_authors._id: 66aca7a3eb69d5b0eba599da
paper_authors.name: Reena Jana
paper_authors._id: 66aca7a3eb69d5b0eba599db
paper_authors.name: Reza Ardeshir Rokni
paper_authors._id: 66aca7a3eb69d5b0eba599dc
paper_authors.name: Rishabh Agarwal
paper_authors._id: 66aca7a3eb69d5b0eba599dd
paper_authors.name: Ryan Mullins
paper_authors._id: 66aca7a3eb69d5b0eba599de
paper_authors.name: Samaneh Saadat
paper_authors._id: 66aca7a3eb69d5b0eba599df
paper_authors.name: Sara Mc Carthy
paper_authors._id: 66aca7a3eb69d5b0eba599e0
paper_authors.name: Sarah Perrin
paper_authors._id: 66aca7a3eb69d5b0eba599e1
paper_authors.name: Sébastien Arnold
paper_authors._id: 66aca7a3eb69d5b0eba599e2
paper_authors.name: Sebastian Krause
paper_authors._id: 66aca7a3eb69d5b0eba599e3
paper_authors.name: Shengyang Dai
paper_authors._id: 66aca7a3eb69d5b0eba599e4
paper_authors.name: Shruti Garg
paper_authors._id: 66aca7a3eb69d5b0eba599e5
paper_authors.name: Shruti Sheth
paper_authors._id: 66aca7a3eb69d5b0eba599e6
paper_authors.name: Sue Ronstrom
paper_authors._id: 66aca7a3eb69d5b0eba599e7
paper_authors.name: Susan Chan
paper_authors._id: 66aca7a3eb69d5b0eba599e8
paper_authors.name: Timothy Jordan
paper_authors._id: 66aca7a3eb69d5b0eba599e9
paper_authors.name: Ting Yu
paper_authors._id: 66aca7a3eb69d5b0eba599ea
paper_authors.name: Tom Eccles
paper_authors._id: 66aca7a3eb69d5b0eba599eb
paper_authors.name: Tom Hennigan
paper_authors._id: 66aca7a3eb69d5b0eba599ec
paper_authors.name: Tomas Kocisky
paper_authors._id: 66aca7a3eb69d5b0eba599ed
paper_authors.name: Tulsee Doshi
paper_authors._id: 66aca7a3eb69d5b0eba599ee
paper_authors.name: Vihan Jain
paper_authors._id: 66aca7a3eb69d5b0eba599ef
paper_authors.name: Vikas Yadav
paper_authors._id: 66aca7a3eb69d5b0eba599f0
paper_authors.name: Vilobh Meshram
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:52:52.733Z
paper_authors._id: 62639745200163ffba79a4b2
paper_authors.avatarUrl: /avatars/587e13f75ec044868d0ec5a5969af60d.svg
paper_authors.fullname: Vilobh Meshram
paper_authors.type: user
paper_authors.user: vilobhmm
paper_authors._id: 66aca7a3eb69d5b0eba599f1
paper_authors.name: Vishal Dharmadhikari
paper_authors._id: 66aca7a3eb69d5b0eba599f2
paper_authors.name: Warren Barkley
paper_authors._id: 66aca7a3eb69d5b0eba599f3
paper_authors.name: Wei Wei
paper_authors._id: 66aca7a3eb69d5b0eba599f4
paper_authors.name: Wenming Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:52:32.891Z
paper_authors._id: 654b22fa2d2fcd6bf2a0dc03
paper_authors.avatarUrl: /avatars/272404d9c935d9b9fbd5fb56fbc85b89.svg
paper_authors.fullname: Wenming Ye
paper_authors.type: user
paper_authors.user: yewboulder
paper_authors._id: 66aca7a3eb69d5b0eba599f5
paper_authors.name: Woohyun Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:52:24.780Z
paper_authors._id: 657b20f3aa07c6e014ba61da
paper_authors.avatarUrl: /avatars/84a59b1dd99116793e1e411a5b44e94b.svg
paper_authors.fullname: Woohyun Han
paper_authors.type: user
paper_authors.user: tink-expo
paper_authors._id: 66aca7a3eb69d5b0eba599f6
paper_authors.name: Woosuk Kwon
paper_authors._id: 66aca7a3eb69d5b0eba599f7
paper_authors.name: Xiang Xu
paper_authors._id: 66aca7a3eb69d5b0eba599f8
paper_authors.name: Zhe Shen
paper_authors._id: 66aca7a3eb69d5b0eba599f9
paper_authors.name: Zhitao Gong
paper_authors._id: 66aca7a3eb69d5b0eba599fa
paper_authors.name: Zichuan Wei
paper_authors._id: 66aca7a3eb69d5b0eba599fb
paper_authors.name: Victor Cotruta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:51:22.088Z
paper_authors._id: 667458e1a5f9723c7699207a
paper_authors.avatarUrl: /avatars/3cfb56465b3e59f6ca87b4458423f6d2.svg
paper_authors.fullname: Victor Cotruta
paper_authors.type: user
paper_authors.user: victorinox25
paper_authors._id: 66aca7a3eb69d5b0eba599fc
paper_authors.name: Phoebe Kirk
paper_authors._id: 66aca7a3eb69d5b0eba599fd
paper_authors.name: Anand Rao
paper_authors._id: 66aca7a3eb69d5b0eba599fe
paper_authors.name: Minh Giang
paper_authors._id: 66aca7a3eb69d5b0eba599ff
paper_authors.name: Ludovic Peran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:50:46.824Z
paper_authors._id: 60e473dcdefbf317ecd88035
paper_authors.avatarUrl: /avatars/e5c069e6cfc73ea5d3d6c2a437490cdd.svg
paper_authors.fullname: LUDOVIC PERAN
paper_authors.type: user
paper_authors.user: LudovicPeran
paper_authors._id: 66aca7a3eb69d5b0eba59a00
paper_authors.name: Tris Warkentin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:50:34.079Z
paper_authors._id: 64d51be85e5f05485ca50751
paper_authors.avatarUrl: /avatars/5802fcc5a5f1e089df41232892f8f502.svg
paper_authors.fullname: Tris Warkentin
paper_authors.type: user
paper_authors.user: trisfromgoogle
paper_authors._id: 66aca7a3eb69d5b0eba59a01
paper_authors.name: Eli Collins
paper_authors._id: 66aca7a3eb69d5b0eba59a02
paper_authors.name: Joelle Barral
paper_authors._id: 66aca7a3eb69d5b0eba59a03
paper_authors.name: Zoubin Ghahramani
paper_authors._id: 66aca7a3eb69d5b0eba59a04
paper_authors.name: Raia Hadsell
paper_authors._id: 66aca7a3eb69d5b0eba59a05
paper_authors.name: D. Sculley
paper_authors._id: 66aca7a3eb69d5b0eba59a06
paper_authors.name: Jeanine Banks
paper_authors._id: 66aca7a3eb69d5b0eba59a07
paper_authors.name: Anca Dragan
paper_authors._id: 66aca7a3eb69d5b0eba59a08
paper_authors.name: Slav Petrov
paper_authors._id: 66aca7a3eb69d5b0eba59a09
paper_authors.name: Oriol Vinyals
paper_authors._id: 66aca7a3eb69d5b0eba59a0a
paper_authors.name: Jeff Dean
paper_authors._id: 66aca7a3eb69d5b0eba59a0b
paper_authors.name: Demis Hassabis
paper_authors._id: 66aca7a3eb69d5b0eba59a0c
paper_authors.name: Koray Kavukcuoglu
paper_authors._id: 66aca7a3eb69d5b0eba59a0d
paper_authors.name: Clement Farabet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:48:30.187Z
paper_authors._id: 638a689ef560ea995b57aad4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670015109688-noauth.jpeg
paper_authors.fullname: Clement Farabet
paper_authors.type: user
paper_authors.user: clmnt
paper_authors._id: 66aca7a3eb69d5b0eba59a0e
paper_authors.name: Elena Buchatskaya
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:48:22.767Z
paper_authors._id: 65da0ecba37e8a543ab732ca
paper_authors.avatarUrl: /avatars/90889950d8c5e0089ada6b3707104a87.svg
paper_authors.fullname: Elena Buchatskaya
paper_authors.type: user
paper_authors.user: ebuchatskaya
paper_authors._id: 66aca7a3eb69d5b0eba59a0f
paper_authors.name: Sebastian Borgeaud
paper_authors._id: 66aca7a3eb69d5b0eba59a10
paper_authors.name: Noah Fiedel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:48:06.968Z
paper_authors._id: 62f82ebd870a3f98bbf9e62a
paper_authors.avatarUrl: /avatars/3b03299e128abf2c8ba82db704855de7.svg
paper_authors.fullname: Noah Fiedel
paper_authors.type: user
paper_authors.user: nfiedel
paper_authors._id: 66aca7a3eb69d5b0eba59a11
paper_authors.name: Armand Joulin
paper_authors._id: 66aca7a3eb69d5b0eba59a12
paper_authors.name: Kathleen Kenealy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:47:54.480Z
paper_authors._id: 666b337c3fc0651814997536
paper_authors.avatarUrl: /avatars/4d6198300c286c5144794bfd2d1fa653.svg
paper_authors.fullname: Kathleen Kenealy
paper_authors.type: user
paper_authors.user: kathleenatgoogle
paper_authors._id: 66aca7a3eb69d5b0eba59a13
paper_authors.name: Robert Dadashi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:47:45.872Z
paper_authors._id: 65fc7c0da57b88a765aea493
paper_authors.avatarUrl: /avatars/440f88769da1113d6158fe7e0514ead3.svg
paper_authors.fullname: Robert Dadashi
paper_authors.type: user
paper_authors.user: ddsh
paper_authors._id: 66aca7a3eb69d5b0eba59a14
paper_authors.name: Alek Andreev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T12:47:36.757Z
paper_authors._id: 65d667858ac5cf899e3b66c4
paper_authors.avatarUrl: /avatars/340468ee8cd9125eceae23a9b8e536a5.svg
paper_authors.fullname: Alek Andreev
paper_authors.type: user
paper_authors.user: alekandreev
paper_title: Gemma 2: Improving Open Language Models at a Practical Size
paper_summary: In this work, we introduce Gemma 2, a new addition to the Gemma family of
  lightweight, state-of-the-art open models, ranging in scale from 2 billion to
  27 billion parameters. In this new version, we apply several known technical
  modifications to the Transformer architecture, such as interleaving
  local-global attentions (Beltagy et al., 2020a) and group-query attention
  (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge
  distillation (Hinton et al., 2015) instead of next token prediction. The
  resulting models deliver the best performance for their size, and even offer
  competitive alternatives to models that are 2-3 times bigger. We release all
  our models to the community.
paper_ai_keywords: Transformer architecture interleaving local-global attentions group-query attention knowledge distillation next token prediction