Gemma 2: Improving Open Language Models at a Practical Size
In this work, we introduce Gemma 2, a new addition to the Gemma family of
lightweight, state-of-the-art open models, ranging in scale from 2 billion to
27 billion parameters. In this new version, we apply several known technical
modifications to the Transformer architecture, such as interleaving
local-global attentions (Beltagy et al., 2020a) and group-query attention
(Ainslie et al., 2023). We also train the 2B and 9B models with knowledge
distillation (Hinton et al., 2015) instead of next token prediction. The
resulting models deliver the best performance for their size, and even offer
competitive alternatives to models that are 2-3 times bigger. We release all
our models to the community.
66aca7a3eb69d5b0eba59951
Gemma Team
66aca7a3eb69d5b0eba59952
Morgane Riviere
admin_assigned
2024-08-02T12:21:44.445Z
66953d345d5615345518ca31
https://cdn-avatars.huggingface.co/v1/production/uploads/66953d345d5615345518ca31/rIpTxlrMxVNB9Td9fJijl.png
Morgane Riviere
user
morugan
66aca7a3eb69d5b0eba59953
Shreya Pathak
admin_assigned
2024-08-02T12:21:51.931Z
667125edfc615a6257be29bc
/avatars/c1b191d6443b7a4ac52866a746ca5717.svg
Shreya Pathak
user
shreyapathak
66aca7a3eb69d5b0eba59954
Pier Giuseppe Sessa
admin_assigned
2024-08-02T12:22:01.722Z
667a7876becec8fc5112cf1f
/avatars/6874f0e5a7edca4145c57d7908f89840.svg
Pier Giuseppe Sessa
user
piergs
66aca7a3eb69d5b0eba59955
Cassidy Hardin
admin_assigned
2024-08-02T12:22:07.910Z
663a5ec5a4152b77b6cf96fd
/avatars/a51901b21d8670e26b2ac57fd482bb63.svg
Cassidy Hardin
user
casshardin
66aca7a3eb69d5b0eba59956
Surya Bhupatiraju
admin_assigned
2024-08-02T12:22:14.718Z
65d7644d19bf1e60614dee6e
/avatars/422a8d28fa9d9ff2c3ec1406352cb346.svg
Surya Bhupatiraju
user
suryabhupa
66aca7a3eb69d5b0eba59957
Léonard Hussenot
admin_assigned
2024-08-02T12:22:24.134Z
667a7b2e1764931987803b46
/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
Léonard Hussenot
user
leonardhussenot
66aca7a3eb69d5b0eba59958
Thomas Mesnard
admin_assigned
2024-08-02T12:22:33.910Z
6639adab8eb8c9c62ac3fb15
/avatars/83133b04922143be1d8a1fb71fb2938f.svg
Thomas Mesnard
user
thomasmesnard
66aca7a3eb69d5b0eba59959
Bobak Shahriari
66aca7a3eb69d5b0eba5995a
Alexandre Ramé
admin_assigned
2024-08-02T12:22:46.308Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
66aca7a3eb69d5b0eba5995b
Johan Ferret
admin_assigned
2024-08-02T12:22:53.748Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
66aca7a3eb69d5b0eba5995c
Peter Liu
66aca7a3eb69d5b0eba5995d
Pouya Tafti
admin_assigned
2024-08-02T12:36:32.170Z
6558091ecafc48de36f848a8
/avatars/3b7e8327d369436d02fe8c58dde036c2.svg
Pouya Tafti
user
pdt
66aca7a3eb69d5b0eba5995e
Abe Friesen
66aca7a3eb69d5b0eba5995f
Michelle Casbon
admin_assigned
2024-08-02T12:36:42.087Z
662806c1167ee93ec8e503a4
/avatars/8dcf47f91328a38833636068686a8a5a.svg
Michelle Casbon
user
michellecasbon
66aca7a3eb69d5b0eba59960
Sabela Ramos
66aca7a3eb69d5b0eba59961
Ravin Kumar
admin_assigned
2024-08-02T12:36:52.026Z
6317476e106740aaad50e2cd
/avatars/7f066cb7b99f24b57f87598bc9ba7417.svg
Ravin Kumar
user
canyon289
66aca7a3eb69d5b0eba59962
Charline Le Lan
66aca7a3eb69d5b0eba59963
Sammy Jerome
66aca7a3eb69d5b0eba59964
Anton Tsitsulin
admin_assigned
2024-08-02T12:37:07.767Z
63ab8e5c8a357d5057ae659e
/avatars/0eed94fcdb6d7c86259bc5b72b5bd41e.svg
Anton Tsitsulin
user
xgfs
66aca7a3eb69d5b0eba59965
Nino Vieillard
66aca7a3eb69d5b0eba59966
Piotr Stanczyk
66aca7a3eb69d5b0eba59967
Sertan Girgin
66aca7a3eb69d5b0eba59968
Nikola Momchev
66aca7a3eb69d5b0eba59969
Matt Hoffman
66aca7a3eb69d5b0eba5996a
Shantanu Thakoor
66aca7a3eb69d5b0eba5996b
Jean-Bastien Grill
66aca7a3eb69d5b0eba5996c
Behnam Neyshabur
66aca7a3eb69d5b0eba5996d
Alanna Walton
66aca7a3eb69d5b0eba5996e
Aliaksei Severyn
admin_assigned
2024-08-02T12:38:40.223Z
66ac94b19c108b7a2af462c5
/avatars/dc1b38b62dc58b8a0f19fd4978f7bb8c.svg
Aliaksei Severyn
user
aseveryn
66aca7a3eb69d5b0eba5996f
Alicia Parrish
admin_assigned
2024-08-02T12:38:46.584Z
659d71eaf0ae1a57ae140831
/avatars/b9eb8c70b4fe5dbd0f1555602ae41b65.svg
Alicia Parrish
user
avparrish
66aca7a3eb69d5b0eba59970
Aliya Ahmad
66aca7a3eb69d5b0eba59971
Allen Hutchison
66aca7a3eb69d5b0eba59972
Alvin Abdagic
admin_assigned
2024-08-02T12:39:19.565Z
66aaa23c08609915b3855f0b
/avatars/6e6c105d921c41477b61af7a4a1b6495.svg
Alvin Abdagic
user
aabdagic
66aca7a3eb69d5b0eba59973
Amanda Carl
66aca7a3eb69d5b0eba59974
Amy Shen
66aca7a3eb69d5b0eba59975
Andy Brock
66aca7a3eb69d5b0eba59976
Andy Coenen
admin_assigned
2024-08-02T12:40:04.549Z
63b334ff922f26a27e79cf1e
/avatars/a3110658bae1a7c9591c096c6ac457f6.svg
Andy Coenen
user
cannoneyed
66aca7a3eb69d5b0eba59977
Anthony Laforge
admin_assigned
2024-08-02T12:40:12.260Z
642cd49253e76b4c22872896
/avatars/ff2bab2077064d16082f0a7592e821ba.svg
Anthony Laforge
user
aglaforge
66aca7a3eb69d5b0eba59978
Antonia Paterson
66aca7a3eb69d5b0eba59979
Ben Bastian
66aca7a3eb69d5b0eba5997a
Bilal Piot
66aca7a3eb69d5b0eba5997b
Bo Wu
66aca7a3eb69d5b0eba5997c
Brandon Royal
admin_assigned
2024-08-02T12:40:39.045Z
64e4f2bfccb9ee90b10ffdba
/avatars/49a28f87b7280c47d2b10cf0fd35d332.svg
Brandon Royal
user
broyal
66aca7a3eb69d5b0eba5997d
Charlie Chen
66aca7a3eb69d5b0eba5997e
Chintu Kumar
66aca7a3eb69d5b0eba5997f
Chris Perry
admin_assigned
2024-08-02T12:41:43.466Z
630cf8e33dc31beba6eeadfc
/avatars/f66eaecb7e710e58f0c06687503c106d.svg
Chris Perry
user
cperry
66aca7a3eb69d5b0eba59980
Chris Welty
66aca7a3eb69d5b0eba59981
Christopher A. Choquette-Choo
admin_assigned
2024-08-02T12:41:55.951Z
649c5307e89a7650158f4965
/avatars/3c71b501db27c86ed686ae26ad1391cd.svg
Christopher A. Choquette-Choo
user
cchoquette
66aca7a3eb69d5b0eba59982
Danila Sinopalnikov
admin_assigned
2024-08-02T12:42:04.581Z
667a6ddc400b4b8da62488f1
/avatars/65772bd92bbbe832eeb51298a231f57a.svg
Danila Sinopalnikov
user
sinopalnikov
66aca7a3eb69d5b0eba59983
David Weinberger
66aca7a3eb69d5b0eba59984
Dimple Vijaykumar
66aca7a3eb69d5b0eba59985
Dominika Rogozińska
66aca7a3eb69d5b0eba59986
Dustin Herbison
66aca7a3eb69d5b0eba59987
Elisa Bandy
66aca7a3eb69d5b0eba59988
Emma Wang
66aca7a3eb69d5b0eba59989
Eric Noland
66aca7a3eb69d5b0eba5998a
Erica Moreira
66aca7a3eb69d5b0eba5998b
Evan Senter
admin_assigned
2024-08-02T12:43:51.964Z
62a47ec0fdd7a13aac264904
/avatars/f2ff23645203ed2399cfe8b447af437a.svg
Evan Senter
user
evansenter
66aca7a3eb69d5b0eba5998c
Evgenii Eltyshev
admin_assigned
2024-08-02T12:43:59.687Z
638b4f0a3bbf29e589029a2e
/avatars/d39fcf06b93affb6d07f534b952c6c16.svg
Evgenii Eltyshev
user
EvgeniiEltyshev
66aca7a3eb69d5b0eba5998d
Francesco Visin
66aca7a3eb69d5b0eba5998e
Gabriel Rasskin
admin_assigned
2024-08-02T12:44:10.319Z
63174c76c92fd6fee319e7f5
/avatars/a2a71b9eed5aa97925740290b2c41dc6.svg
Gabriel Rasskin
user
grasskin
66aca7a3eb69d5b0eba5998f
Gary Wei
admin_assigned
2024-08-02T12:44:43.648Z
66a138cc627519df4aa4b7ab
https://cdn-avatars.huggingface.co/v1/production/uploads/66a138cc627519df4aa4b7ab/XdIE8H9v22kvsWweMDF6d.jpeg
Gary Wei
user
cdwei-gg
66aca7a3eb69d5b0eba59990
Glenn Cameron
66aca7a3eb69d5b0eba59991
Gus Martins
66aca7a3eb69d5b0eba59992
Hadi Hashemi
admin_assigned
2024-08-02T12:45:20.378Z
64da4f7578864cd2221af0c1
/avatars/428d3fd717654893984f9d01f3d6667d.svg
Hadi Hashemi
user
hadi-hashemi
66aca7a3eb69d5b0eba59993
Hanna Klimczak-Plucińska
66aca7a3eb69d5b0eba59994
Harleen Batra
66aca7a3eb69d5b0eba59995
Harsh Dhand
66aca7a3eb69d5b0eba59996
Ivan Nardini
admin_assigned
2024-08-02T12:45:56.728Z
6280f4b052815a0dc7399902
/avatars/e098c20e16d17e687d5d0c78745cdc7d.svg
Ivan Nardini
user
IlNardo92
66aca7a3eb69d5b0eba59997
Jacinda Mein
66aca7a3eb69d5b0eba59998
Jack Zhou
66aca7a3eb69d5b0eba59999
James Svensson
admin_assigned
2024-08-02T12:46:11.238Z
62d33483c20456076a2a2c06
/avatars/e1ac677101381030a3174cd859604afa.svg
James Svensson
user
Sven84
66aca7a3eb69d5b0eba5999a
Jeff Stanway
66aca7a3eb69d5b0eba5999b
Jetha Chan
admin_assigned
2024-08-02T12:46:31.465Z
6670e8966d34fe6262885205
/avatars/f5f7845a54c2bf8e418989f113272578.svg
Jetha Chan
user
jethac
66aca7a3eb69d5b0eba5999c
Jin Zhou
66aca7a3eb69d5b0eba5999d
Joana Carrasqueira
66aca7a3eb69d5b0eba5999e
Joana Iljazi
66aca7a3eb69d5b0eba5999f
Jocelyn Becker
66aca7a3eb69d5b0eba599a0
Joe Fernandez
66aca7a3eb69d5b0eba599a1
Joost van Amersfoort
66aca7a3eb69d5b0eba599a2
Josh Gordon
66aca7a3eb69d5b0eba599a3
Josh Lipschultz
66aca7a3eb69d5b0eba599a4
Josh Newlan
66aca7a3eb69d5b0eba599a5
Ju-yeong Ji
66aca7a3eb69d5b0eba599a6
Kareem Mohamed
66aca7a3eb69d5b0eba599a7
Kartikeya Badola
66aca7a3eb69d5b0eba599a8
Kat Black
66aca7a3eb69d5b0eba599a9
Katie Millican
66aca7a3eb69d5b0eba599aa
Keelin McDonell
admin_assigned
2024-08-02T12:56:57.135Z
62b9e7c04dde8074b78be449
https://cdn-avatars.huggingface.co/v1/production/uploads/1675986645321-62b9e7c04dde8074b78be449.jpeg
Keelin McDonell
user
keelin
66aca7a3eb69d5b0eba599ab
Kelvin Nguyen
66aca7a3eb69d5b0eba599ac
Kiranbir Sodhia
66aca7a3eb69d5b0eba599ad
Kish Greene
66aca7a3eb69d5b0eba599ae
Lars Lowe Sjoesund
66aca7a3eb69d5b0eba599af
Lauren Usui
admin_assigned
2024-08-02T13:00:24.875Z
63dd3ddea8877129a15a534c
/avatars/96bdadf176162856d1687f8298915484.svg
lauren usui
user
lusui
66aca7a3eb69d5b0eba599b0
Laurent Sifre
66aca7a3eb69d5b0eba599b1
Lena Heuermann
66aca7a3eb69d5b0eba599b2
Leticia Lago
66aca7a3eb69d5b0eba599b3
Lilly McNealus
66aca7a3eb69d5b0eba599b4
Livio Baldini Soares
66aca7a3eb69d5b0eba599b5
Logan Kilpatrick
66aca7a3eb69d5b0eba599b6
Lucas Dixon
admin_assigned
2024-08-02T12:57:55.078Z
64c25a9c2b1ffc2eecbce31d
https://cdn-avatars.huggingface.co/v1/production/uploads/64c25a9c2b1ffc2eecbce31d/uTp3yOPdmCSAmhzWwKUjK.png
Lucas Dixon
user
iislucas
66aca7a3eb69d5b0eba599b7
Luciano Martins
66aca7a3eb69d5b0eba599b8
Machel Reid
66aca7a3eb69d5b0eba599b9
Manvinder Singh
admin_assigned
2024-08-02T12:57:17.021Z
64dc01ab3764a5137e8fbd70
/avatars/807bef46ed74bbcc6bd9cac9b9acf82b.svg
Manvinder Singh
user
manvinder
66aca7a3eb69d5b0eba599ba
Mark Iverson
66aca7a3eb69d5b0eba599bb
Martin Görner
66aca7a3eb69d5b0eba599bc
Mat Velloso
66aca7a3eb69d5b0eba599bd
Mateo Wirth
66aca7a3eb69d5b0eba599be
Matt Davidow
66aca7a3eb69d5b0eba599bf
Matt Miller
66aca7a3eb69d5b0eba599c0
Matthew Rahtz
admin_assigned
2024-08-02T13:00:57.136Z
6691892b389abe9d474d80da
/avatars/7b2fa384c7f1fcaef1e99d297b9a624b.svg
Matthew Rahtz
user
gdm-mrahtz
66aca7a3eb69d5b0eba599c1
Matthew Watson
66aca7a3eb69d5b0eba599c2
Meg Risdal
admin_assigned
2024-08-02T13:00:40.049Z
626b23fc822e3d85324babed
https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
Meg Risdal
user
megrisdal
66aca7a3eb69d5b0eba599c3
Mehran Kazemi
66aca7a3eb69d5b0eba599c4
Michael Moynihan
admin_assigned
2024-08-02T13:01:19.758Z
65aeae63563e362f5521e32a
/avatars/ad3a9f68cae8cbaf639084e31adffc54.svg
Michael Moynihan
user
mmoynihan
66aca7a3eb69d5b0eba599c5
Ming Zhang
66aca7a3eb69d5b0eba599c6
Minsuk Kahng
admin_assigned
2024-08-02T13:23:27.543Z
6301f3fb4ccccaa23d38fb7f
/avatars/4226111991d1788d3d442ffb2f33ea48.svg
Minsuk Kahng
user
minsukkahng
66aca7a3eb69d5b0eba599c7
Minwoo Park
66aca7a3eb69d5b0eba599c8
Mofi Rahman
admin_assigned
2024-08-02T13:01:42.741Z
65d5f883ba0ad765ea02bb9c
https://cdn-avatars.huggingface.co/v1/production/uploads/65d5f883ba0ad765ea02bb9c/Z0lPfLx6p4i-NAyCslZkN.png
mofi rahman
user
mofirahman
66aca7a3eb69d5b0eba599c9
Mohit Khatwani
66aca7a3eb69d5b0eba599ca
Natalie Dao
66aca7a3eb69d5b0eba599cb
Nenshad Bardoliwalla
admin_assigned
2024-08-02T12:58:17.909Z
65d618051a95fdcf7c59086d
/avatars/05256144ae27b46b0bd2ff4eb0ea6642.svg
Nenshad Bardoliwalla
user
nenshadbardoliwalla
66aca7a3eb69d5b0eba599cc
Nesh Devanathan
66aca7a3eb69d5b0eba599cd
Neta Dumai
66aca7a3eb69d5b0eba599ce
Nilay Chauhan
admin_assigned
2024-08-02T13:00:17.063Z
66a7935b7959dfd1e912afca
/avatars/3d31bfeaf0c008b2c9dae99b8d0298b7.svg
Nilay Chauhan
user
nilayc
66aca7a3eb69d5b0eba599cf
Oscar Wahltinez
66aca7a3eb69d5b0eba599d0
Pankil Botarda
66aca7a3eb69d5b0eba599d1
Parker Barnes
66aca7a3eb69d5b0eba599d2
Paul Barham
66aca7a3eb69d5b0eba599d3
Paul Michel
66aca7a3eb69d5b0eba599d4
Pengchong Jin
66aca7a3eb69d5b0eba599d5
Petko Georgiev
66aca7a3eb69d5b0eba599d6
Phil Culliton
claimed_verified
2025-05-29T20:18:12.540Z
627a9aa0f967ef475abe8037
/avatars/47cf94d6c015577d0a860b617c9ea2b8.svg
Phil Culliton
user
PhilCulliton
66aca7a3eb69d5b0eba599d7
Pradeep Kuppala
66aca7a3eb69d5b0eba599d8
Ramona Comanescu
66aca7a3eb69d5b0eba599d9
Ramona Merhej
66aca7a3eb69d5b0eba599da
Reena Jana
66aca7a3eb69d5b0eba599db
Reza Ardeshir Rokni
66aca7a3eb69d5b0eba599dc
Rishabh Agarwal
66aca7a3eb69d5b0eba599dd
Ryan Mullins
claimed_verified
2025-03-28T08:39:13.010Z
65a402a53522df7a27125823
https://cdn-avatars.huggingface.co/v1/production/uploads/65a402a53522df7a27125823/tVR6lZlKmjPqi7dTZBGoS.jpeg
Ryan Mullins
user
RyanMullins
66aca7a3eb69d5b0eba599de
Samaneh Saadat
66aca7a3eb69d5b0eba599df
Sara Mc Carthy
66aca7a3eb69d5b0eba599e0
Sarah Perrin
66aca7a3eb69d5b0eba599e1
Sébastien Arnold
66aca7a3eb69d5b0eba599e2
Sebastian Krause
66aca7a3eb69d5b0eba599e3
Shengyang Dai
66aca7a3eb69d5b0eba599e4
Shruti Garg
66aca7a3eb69d5b0eba599e5
Shruti Sheth
66aca7a3eb69d5b0eba599e6
Sue Ronstrom
66aca7a3eb69d5b0eba599e7
Susan Chan
66aca7a3eb69d5b0eba599e8
Timothy Jordan
66aca7a3eb69d5b0eba599e9
Ting Yu
66aca7a3eb69d5b0eba599ea
Tom Eccles
66aca7a3eb69d5b0eba599eb
Tom Hennigan
66aca7a3eb69d5b0eba599ec
Tomas Kocisky
66aca7a3eb69d5b0eba599ed
Tulsee Doshi
66aca7a3eb69d5b0eba599ee
Vihan Jain
66aca7a3eb69d5b0eba599ef
Vikas Yadav
66aca7a3eb69d5b0eba599f0
Vilobh Meshram
admin_assigned
2024-08-02T12:52:52.733Z
62639745200163ffba79a4b2
/avatars/587e13f75ec044868d0ec5a5969af60d.svg
Vilobh Meshram
user
vilobhmm
66aca7a3eb69d5b0eba599f1
Vishal Dharmadhikari
66aca7a3eb69d5b0eba599f2
Warren Barkley
66aca7a3eb69d5b0eba599f3
Wei Wei
66aca7a3eb69d5b0eba599f4
Wenming Ye
admin_assigned
2024-08-02T12:52:32.891Z
654b22fa2d2fcd6bf2a0dc03
/avatars/272404d9c935d9b9fbd5fb56fbc85b89.svg
Wenming Ye
user
yewboulder
66aca7a3eb69d5b0eba599f5
Woohyun Han
admin_assigned
2024-08-02T12:52:24.780Z
657b20f3aa07c6e014ba61da
/avatars/84a59b1dd99116793e1e411a5b44e94b.svg
Woohyun Han
user
tink-expo
66aca7a3eb69d5b0eba599f6
Woosuk Kwon
66aca7a3eb69d5b0eba599f7
Xiang Xu
66aca7a3eb69d5b0eba599f8
Zhe Shen
66aca7a3eb69d5b0eba599f9
Zhitao Gong
66aca7a3eb69d5b0eba599fa
Zichuan Wei
66aca7a3eb69d5b0eba599fb
Victor Cotruta
admin_assigned
2024-08-02T12:51:22.088Z
667458e1a5f9723c7699207a
/avatars/3cfb56465b3e59f6ca87b4458423f6d2.svg
Victor Cotruta
user
victorinox25
66aca7a3eb69d5b0eba599fc
Phoebe Kirk
66aca7a3eb69d5b0eba599fd
Anand Rao
66aca7a3eb69d5b0eba599fe
Minh Giang
66aca7a3eb69d5b0eba599ff
Ludovic Peran
admin_assigned
2024-08-02T12:50:46.824Z
60e473dcdefbf317ecd88035
/avatars/e5c069e6cfc73ea5d3d6c2a437490cdd.svg
LUDOVIC PERAN
user
LudovicPeran
66aca7a3eb69d5b0eba59a00
Tris Warkentin
admin_assigned
2024-08-02T12:50:34.079Z
64d51be85e5f05485ca50751
/avatars/5802fcc5a5f1e089df41232892f8f502.svg
Tris Warkentin
user
trisfromgoogle
66aca7a3eb69d5b0eba59a01
Eli Collins
66aca7a3eb69d5b0eba59a02
Joelle Barral
66aca7a3eb69d5b0eba59a03
Zoubin Ghahramani
66aca7a3eb69d5b0eba59a04
Raia Hadsell
66aca7a3eb69d5b0eba59a05
D. Sculley
66aca7a3eb69d5b0eba59a06
Jeanine Banks
66aca7a3eb69d5b0eba59a07
Anca Dragan
66aca7a3eb69d5b0eba59a08
Slav Petrov
66aca7a3eb69d5b0eba59a09
Oriol Vinyals
66aca7a3eb69d5b0eba59a0a
Jeff Dean
66aca7a3eb69d5b0eba59a0b
Demis Hassabis
66aca7a3eb69d5b0eba59a0c
Koray Kavukcuoglu
66aca7a3eb69d5b0eba59a0d
Clement Farabet
admin_assigned
2024-08-02T12:48:30.187Z
638a689ef560ea995b57aad4
https://cdn-avatars.huggingface.co/v1/production/uploads/1670015109688-noauth.jpeg
Clement Farabet
user
clmnt
66aca7a3eb69d5b0eba59a0e
Elena Buchatskaya
admin_assigned
2024-08-02T12:48:22.767Z
65da0ecba37e8a543ab732ca
/avatars/90889950d8c5e0089ada6b3707104a87.svg
Elena Buchatskaya
user
ebuchatskaya
66aca7a3eb69d5b0eba59a0f
Sebastian Borgeaud
66aca7a3eb69d5b0eba59a10
Noah Fiedel
admin_assigned
2024-08-02T12:48:06.968Z
62f82ebd870a3f98bbf9e62a
/avatars/3b03299e128abf2c8ba82db704855de7.svg
Noah Fiedel
user
nfiedel
66aca7a3eb69d5b0eba59a11
Armand Joulin
66aca7a3eb69d5b0eba59a12
Kathleen Kenealy
admin_assigned
2024-08-02T12:47:54.480Z
666b337c3fc0651814997536
/avatars/4d6198300c286c5144794bfd2d1fa653.svg
Kathleen Kenealy
user
kathleenatgoogle
66aca7a3eb69d5b0eba59a13
Robert Dadashi
admin_assigned
2024-08-02T12:47:45.872Z
65fc7c0da57b88a765aea493
/avatars/440f88769da1113d6158fe7e0514ead3.svg
Robert Dadashi
user
ddsh
66aca7a3eb69d5b0eba59a14
Alek Andreev
admin_assigned
2024-08-02T12:47:36.757Z
65d667858ac5cf899e3b66c4
/avatars/340468ee8cd9125eceae23a9b8e536a5.svg
Alek Andreev
user
alekandreev
Gemma 2: Improving Open Language Models at a Practical Size
In this work, we introduce Gemma 2, a new addition to the Gemma family of
lightweight, state-of-the-art open models, ranging in scale from 2 billion to
27 billion parameters. In this new version, we apply several known technical
modifications to the Transformer architecture, such as interleaving
local-global attentions (Beltagy et al., 2020a) and group-query attention
(Ainslie et al., 2023). We also train the 2B and 9B models with knowledge
distillation (Hinton et al., 2015) instead of next token prediction. The
resulting models deliver the best performance for their size, and even offer
competitive alternatives to models that are 2-3 times bigger. We release all
our models to the community.
Gemma 2 introduces improvements in the Transformer architecture through interleaving local-global attentions and group-query attention, showcasing superior performance relative to its size.
Transformer architecture interleaving local-global attentions group-query attention knowledge distillation