title: StarCoder 2 and The Stack v2: The Next Generation
summary: The BigCode project, an open-scientific collaboration focused on the
  responsible development of Large Language Models for Code (Code LLMs),
  introduces StarCoder2. In partnership with Software Heritage (SWH), we build
  The Stack v2 on top of the digital commons of their source code archive.
  Alongside the SWH repositories spanning 619 programming languages, we carefully
  select other high-quality data sources, such as GitHub pull requests, Kaggle
  notebooks, and code documentation. This results in a training set that is 4x
  larger than the first StarCoder dataset. We train StarCoder2 models with 3B,
  7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate
  them on a comprehensive set of Code LLM benchmarks. We find that our small
  model, StarCoder2-3B, outperforms other Code LLMs of similar size on most
  benchmarks, and also outperforms StarCoderBase-15B. Our large model,
  StarCoder2- 15B, significantly outperforms other models of comparable size. In
  addition, it matches or outperforms CodeLlama-34B, a model more than twice its
  size. Although DeepSeekCoder- 33B is the best-performing model at code
  completion for high-resource languages, we find that StarCoder2-15B outperforms
  it on math and code reasoning benchmarks, as well as several low-resource
  languages. We make the model weights available under an OpenRAIL license and
  ensure full transparency regarding the training data by releasing the SoftWare
  Heritage persistent IDentifiers (SWHIDs) of the source code data.
paper_authors._id: 65e161d606c9c86f25225a23
paper_authors.name: Anton Lozhkov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:58:06.843Z
paper_authors._id: 602e6dee60e3dd96631c906e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png
paper_authors.fullname: Anton Lozhkov
paper_authors.type: user
paper_authors.user: anton-l
paper_authors._id: 65e161d606c9c86f25225a24
paper_authors.name: Raymond Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:58:23.179Z
paper_authors._id: 62d9a30cbfc8699e098791c5
paper_authors.avatarUrl: /avatars/c80cf773f436748c992d76ca042dd5fb.svg
paper_authors.fullname: Raymond Li
paper_authors.type: user
paper_authors.user: RaymondLi
paper_authors._id: 65e161d606c9c86f25225a25
paper_authors.name: Loubna Ben Allal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:55:19.844Z
paper_authors._id: 61c141342aac764ce1654e43
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
paper_authors.fullname: Loubna Ben Allal
paper_authors.type: user
paper_authors.user: loubnabnl
paper_authors._id: 65e161d606c9c86f25225a26
paper_authors.name: Federico Cassano
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:58:30.709Z
paper_authors._id: 642ca13cc3684e5a4e806661
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642ca13cc3684e5a4e806661/ELhdKsK429zi4wZkVkx9Y.jpeg
paper_authors.fullname: Federico Cassano
paper_authors.type: user
paper_authors.user: cassanof
paper_authors._id: 65e161d606c9c86f25225a27
paper_authors.name: Joel Lamy-Poirier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:58:50.211Z
paper_authors._id: 6372ec0e759c1ca82045c43d
paper_authors.avatarUrl: /avatars/86d5dbdb10a0cbf9a39a32e2d87361b6.svg
paper_authors.fullname: Joel Lamy-Poirier
paper_authors.type: user
paper_authors.user: jlamypoirier
paper_authors._id: 65e161d606c9c86f25225a28
paper_authors.name: Nouamane Tazi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:58:56.576Z
paper_authors._id: 5ff8c9f4b2035d9a81a859f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652134289581-5ff8c9f4b2035d9a81a859f7.jpeg
paper_authors.fullname: Nouamane Tazi
paper_authors.type: user
paper_authors.user: nouamanetazi
paper_authors._id: 65e161d606c9c86f25225a29
paper_authors.name: Ao Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:59:15.123Z
paper_authors._id: 645a5024e7e8a4b57bd8e73e
paper_authors.avatarUrl: /avatars/6cfbf7138ad554dfdb8052d5edc7d320.svg
paper_authors.fullname: Ao Tang
paper_authors.type: user
paper_authors.user: thhaus
paper_authors._id: 65e161d606c9c86f25225a2a
paper_authors.name: Dmytro Pykhtar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:59:35.269Z
paper_authors._id: 62e12c61cb1f164f2cb0afd3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658924005273-noauth.jpeg
paper_authors.fullname: Dmytro Pykhtar
paper_authors.type: user
paper_authors.user: dpykhtar
paper_authors._id: 65e161d606c9c86f25225a2b
paper_authors.name: Jiawei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:46.334Z
paper_authors._id: 644b584a9279988e0cbeb664
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644b584a9279988e0cbeb664/fz4pn7HeXXfpvz23v8E8t.png
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: ganler
paper_authors._id: 65e161d606c9c86f25225a2c
paper_authors.name: Yuxiang Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:00:10.606Z
paper_authors._id: 632a176259950c1d279d5ea7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
paper_authors.fullname: Yuxiang Wei
paper_authors.type: user
paper_authors.user: yuxiang630
paper_authors._id: 65e161d606c9c86f25225a2d
paper_authors.name: Tianyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:26:02.046Z
paper_authors._id: 629e2bcc46b4826be2c57fe3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629e2bcc46b4826be2c57fe3/41BiA52XlZi31ABsljFiq.jpeg
paper_authors.fullname: Tianyang Liu
paper_authors.type: user
paper_authors.user: tianyang
paper_authors._id: 65e161d606c9c86f25225a2e
paper_authors.name: Max Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:00:24.947Z
paper_authors._id: 633fe49ff81b9d10135fe20a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633fe49ff81b9d10135fe20a/3s7dJJgF6jnS-DypXXybC.png
paper_authors.fullname: Max Tian
paper_authors.type: user
paper_authors.user: maxmatical
paper_authors._id: 65e161d606c9c86f25225a2f
paper_authors.name: Denis Kocetkov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:00:31.401Z
paper_authors._id: 62dea51feee79ca5ef8f834a
paper_authors.avatarUrl: /avatars/17e9356625785622762874fe96900599.svg
paper_authors.fullname: Denis Kocetkov
paper_authors.type: user
paper_authors.user: denisko
paper_authors._id: 65e161d606c9c86f25225a30
paper_authors.name: Arthur Zucker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:00:44.997Z
paper_authors._id: 62441cb7456803e95009a08f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg
paper_authors.fullname: Arthur Zucker
paper_authors.type: user
paper_authors.user: ArthurZ
paper_authors._id: 65e161d606c9c86f25225a31
paper_authors.name: Younes Belkada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:00:53.289Z
paper_authors._id: 62441d1d9fdefb55a0b7d12c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
paper_authors.fullname: Younes Belkada
paper_authors.type: user
paper_authors.user: ybelkada
paper_authors._id: 65e161d606c9c86f25225a32
paper_authors.name: Zijian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:01:21.399Z
paper_authors._id: 6465183934acce85aea2c3b3
paper_authors.avatarUrl: /avatars/275a0811381afc980d339ba2aa1294c7.svg
paper_authors.fullname: Zijian Wang
paper_authors.type: user
paper_authors.user: zijwang
paper_authors._id: 65e161d606c9c86f25225a33
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:56.592Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 65e161d606c9c86f25225a34
paper_authors.name: Dmitry Abulkhanov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:01:30.784Z
paper_authors._id: 636a9a07e3ad78bc68b1a5a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668020490988-636a9a07e3ad78bc68b1a5a2.jpeg
paper_authors.fullname: Dmitry Abulkhanov
paper_authors.type: user
paper_authors.user: mponty
paper_authors._id: 65e161d606c9c86f25225a35
paper_authors.name: Indraneil Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:01:36.838Z
paper_authors._id: 62054c0b522e40b4a18d8744
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62054c0b522e40b4a18d8744/e6bJyJCwEuOZI8BiKC0rx.jpeg
paper_authors.fullname: Indraneil Paul
paper_authors.type: user
paper_authors.user: iNeil77
paper_authors._id: 65e161d606c9c86f25225a36
paper_authors.name: Zhuang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:02:06.177Z
paper_authors._id: 63d159132036e44c44f87a91
paper_authors.avatarUrl: /avatars/6a2c9e5b3b25cf1949277d8c40c0070b.svg
paper_authors.fullname: Zhuang Li
paper_authors.type: user
paper_authors.user: lizhuang144
paper_authors._id: 65e161d606c9c86f25225a37
paper_authors.name: Wen-Ding Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:02:13.737Z
paper_authors._id: 62e221dfcb1f164f2cb8a66b
paper_authors.avatarUrl: /avatars/06f05622e232304d3f0b8c291f3263be.svg
paper_authors.fullname: Wen-Ding Li
paper_authors.type: user
paper_authors.user: xu3kev
paper_authors._id: 65e161d606c9c86f25225a38
paper_authors.name: Megan Risdal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:40:55.577Z
paper_authors._id: 626b23fc822e3d85324babed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
paper_authors.fullname: Meg Risdal
paper_authors.type: user
paper_authors.user: megrisdal
paper_authors._id: 65e161d606c9c86f25225a39
paper_authors.name: Jia Li
paper_authors._id: 65e161d606c9c86f25225a3a
paper_authors.name: Jian Zhu
paper_authors._id: 65e161d606c9c86f25225a3b
paper_authors.name: Terry Yue Zhuo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:44.067Z
paper_authors._id: 62b7fb545233925f253531c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
paper_authors.fullname: Terry Yue Zhuo
paper_authors.type: user
paper_authors.user: terryyz
paper_authors._id: 65e161d606c9c86f25225a3c
paper_authors.name: Evgenii Zheltonozhskii
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:02:38.479Z
paper_authors._id: 61ae250602b8655e3c8973cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61ae250602b8655e3c8973cd/aXp22-B-oKlm_h6kLx9Uv.jpeg
paper_authors.fullname: Evgenii Zheltonozhskii
paper_authors.type: user
paper_authors.user: EvgeniyZh
paper_authors._id: 65e161d606c9c86f25225a3d
paper_authors.name: Nii Osae Osae Dade
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:44.709Z
paper_authors._id: 621d2948962e5162b2f7d985
paper_authors.avatarUrl: /avatars/545485b091e96efb64ffa67baa89ccd9.svg
paper_authors.fullname: Nii Osae
paper_authors.type: user
paper_authors.user: starkosae
paper_authors._id: 65e161d606c9c86f25225a3e
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:03:08.512Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 65e161d606c9c86f25225a3f
paper_authors.name: Lucas Krauß
paper_authors._id: 65e161d606c9c86f25225a40
paper_authors.name: Naman Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:56.924Z
paper_authors._id: 6411ed451e42164b9f0f8b24
paper_authors.avatarUrl: /avatars/a5c023f407894d3d1dba7b343b847380.svg
paper_authors.fullname: Naman Jain
paper_authors.type: user
paper_authors.user: StringChaos
paper_authors._id: 65e161d606c9c86f25225a41
paper_authors.name: Yixuan Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:04:30.527Z
paper_authors._id: 6144c40da8f4f7baca305243
paper_authors.avatarUrl: /avatars/1f9bbb70f859f34563fdd0a7de4cd254.svg
paper_authors.fullname: Yixuan Su
paper_authors.type: user
paper_authors.user: pangpang666
paper_authors._id: 65e161d606c9c86f25225a42
paper_authors.name: Xuanli He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:32:16.704Z
paper_authors._id: 614b0ab851b6682b9e56fd92
paper_authors.avatarUrl: /avatars/81606ae85106b89a34bd08283646d07b.svg
paper_authors.fullname: Xuanli He
paper_authors.type: user
paper_authors.user: zodiache
paper_authors._id: 65e161d606c9c86f25225a43
paper_authors.name: Manan Dey
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:32:24.746Z
paper_authors._id: 5fc53305fa6eef7667a4d691
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1623089209793-5fc53305fa6eef7667a4d691.jpeg
paper_authors.fullname: Manan Dey
paper_authors.type: user
paper_authors.user: manandey
paper_authors._id: 65e161d606c9c86f25225a44
paper_authors.name: Edoardo Abati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:32:32.918Z
paper_authors._id: 61be0c0394bd578f72861849
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655399022242-61be0c0394bd578f72861849.jpeg
paper_authors.fullname: Edoardo Abati
paper_authors.type: user
paper_authors.user: EdoAbati
paper_authors._id: 65e161d606c9c86f25225a45
paper_authors.name: Yekun Chai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-04T17:20:25.292Z
paper_authors._id: 626a6ce8adf559c88de9ace0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651142275247-626a6ce8adf559c88de9ace0.jpeg
paper_authors.fullname: CYK
paper_authors.type: user
paper_authors.user: cyk1337
paper_authors._id: 65e161d606c9c86f25225a46
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:32:51.477Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 65e161d606c9c86f25225a47
paper_authors.name: Xiangru Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:32:58.810Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 65e161d606c9c86f25225a48
paper_authors.name: Muhtasham Oblokulov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:50.531Z
paper_authors._id: 5fcaabed246881afd5b00167
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651847561574-5fcaabed246881afd5b00167.jpeg
paper_authors.fullname: Muhtasham Oblokulov
paper_authors.type: user
paper_authors.user: muhtasham
paper_authors._id: 65e161d606c9c86f25225a49
paper_authors.name: Christopher Akiki
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:48.537Z
paper_authors._id: 5e70f6048ce3c604d78fe133
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1646492542174-5e70f6048ce3c604d78fe133.jpeg
paper_authors.fullname: Christopher Akiki
paper_authors.type: user
paper_authors.user: christopher
paper_authors._id: 65e161d606c9c86f25225a4a
paper_authors.name: Marc Marone
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:33:05.734Z
paper_authors._id: 63e410e88083f19a218be964
paper_authors.avatarUrl: /avatars/ce271d1686f6e5ee1f5b2d429cb60de6.svg
paper_authors.fullname: Marc Marone
paper_authors.type: user
paper_authors.user: mmarone
paper_authors._id: 65e161d606c9c86f25225a4b
paper_authors.name: Chenghao Mou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:33:14.231Z
paper_authors._id: 607a5b44489fc71534e91c0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1638743128077-607a5b44489fc71534e91c0e.jpeg
paper_authors.fullname: Chenghao Mou
paper_authors.type: user
paper_authors.user: chenghao
paper_authors._id: 65e161d606c9c86f25225a4c
paper_authors.name: Mayank Mishra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:54.340Z
paper_authors._id: 62cd5057674cdb524450093d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
paper_authors.fullname: Mayank Mishra
paper_authors.type: user
paper_authors.user: mayank-mishra
paper_authors._id: 65e161d606c9c86f25225a4d
paper_authors.name: Alex Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:46.853Z
paper_authors._id: 6179abf8ec6ce4dc2e5f2376
paper_authors.avatarUrl: /avatars/d10c6a1b350146b36949a24220471295.svg
paper_authors.fullname: Alex Gu
paper_authors.type: user
paper_authors.user: minimario
paper_authors._id: 65e161d606c9c86f25225a4e
paper_authors.name: Binyuan Hui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:58:41.147Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 65e161d606c9c86f25225a4f
paper_authors.name: Tri Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:33:39.454Z
paper_authors._id: 64b8a6b5cf14c2fabe98159b
paper_authors.avatarUrl: /avatars/dbc009451865435bf290791beadc4723.svg
paper_authors.fullname: Tri Dao
paper_authors.type: user
paper_authors.user: tridao
paper_authors._id: 65e161d606c9c86f25225a50
paper_authors.name: Armel Zebaze
paper_authors._id: 65e161d606c9c86f25225a51
paper_authors.name: Olivier Dehaene
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:33:55.167Z
paper_authors._id: 62a093d63e7d1dda047039fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a093d63e7d1dda047039fc/QGpVSKuJLwl2EsiffCYML.jpeg
paper_authors.fullname: Olivier Dehaene
paper_authors.type: user
paper_authors.user: olivierdehaene
paper_authors._id: 65e161d606c9c86f25225a52
paper_authors.name: Nicolas Patry
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:34:17.991Z
paper_authors._id: 5e2967b819407e3277369b95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png
paper_authors.fullname: Nicolas Patry
paper_authors.type: user
paper_authors.user: Narsil
paper_authors._id: 65e161d606c9c86f25225a53
paper_authors.name: Canwen Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:34:26.865Z
paper_authors._id: 5e3c29d8f55e2b62848a5224
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618318588221-5e3c29d8f55e2b62848a5224.png
paper_authors.fullname: Canwen Xu
paper_authors.type: user
paper_authors.user: canwenxu
paper_authors._id: 65e161d606c9c86f25225a54
paper_authors.name: Julian McAuley
paper_authors._id: 65e161d606c9c86f25225a55
paper_authors.name: Han Hu
paper_authors._id: 65e161d606c9c86f25225a56
paper_authors.name: Torsten Scholak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:34:47.627Z
paper_authors._id: 60ecaa5efee13fee7ada7af4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
paper_authors.fullname: Torsten Scholak
paper_authors.type: user
paper_authors.user: tscholak
paper_authors._id: 65e161d606c9c86f25225a57
paper_authors.name: Sebastien Paquet
paper_authors._id: 65e161d606c9c86f25225a58
paper_authors.name: Jennifer Robinson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:09.172Z
paper_authors._id: 6452f03dbeba452314fe59fa
paper_authors.avatarUrl: /avatars/ed7e760b93e2a701bfbcf1ecc62ac33c.svg
paper_authors.fullname: Jennifer Robinson
paper_authors.type: user
paper_authors.user: Jenniferrobinsonservicenow
paper_authors._id: 65e161d606c9c86f25225a59
paper_authors.name: Carolyn Jane Anderson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:19.548Z
paper_authors._id: 6243199444c9c3b21be74c50
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6243199444c9c3b21be74c50/uxnj03NBcV_TeC3eV2U-Q.jpeg
paper_authors.fullname: Carolyn Anderson
paper_authors.type: user
paper_authors.user: canders1
paper_authors._id: 65e161d606c9c86f25225a5a
paper_authors.name: Nicolas Chapados
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:27.692Z
paper_authors._id: 631f54aa5ba8c026340b13cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
paper_authors.fullname: Nicolas Chapados
paper_authors.type: user
paper_authors.user: nicolaschapados
paper_authors._id: 65e161d606c9c86f25225a5b
paper_authors.name: Mostofa Patwary
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:35.319Z
paper_authors._id: 630544b09d2531fabd156fd3
paper_authors.avatarUrl: /avatars/7b5374244a887577834fb4524ff76d01.svg
paper_authors.fullname: Mostofa Patwary
paper_authors.type: user
paper_authors.user: mpatwary
paper_authors._id: 65e161d606c9c86f25225a5c
paper_authors.name: Nima Tajbakhsh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:42.399Z
paper_authors._id: 6542c90c4f89f533be7609bd
paper_authors.avatarUrl: /avatars/9f5ee3709940fe9632c3896be79796a6.svg
paper_authors.fullname: Nima Tajbakhsh
paper_authors.type: user
paper_authors.user: ntajbakhsh
paper_authors._id: 65e161d606c9c86f25225a5d
paper_authors.name: Yacine Jernite
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:50.209Z
paper_authors._id: 5ee3a7cd2a3eae3cbdad1305
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594144055859-5ee3a7cd2a3eae3cbdad1305.jpeg
paper_authors.fullname: Yacine Jernite
paper_authors.type: user
paper_authors.user: yjernite
paper_authors._id: 65e161d606c9c86f25225a5e
paper_authors.name: Carlos Muñoz Ferrandis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:35:57.932Z
paper_authors._id: 629f05bddaca51e26a7cfef1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654588848873-noauth.jpeg
paper_authors.fullname: Carlos Muñoz Ferrandis
paper_authors.type: user
paper_authors.user: CarlosMF
paper_authors._id: 65e161d606c9c86f25225a5f
paper_authors.name: Lingming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:36:05.659Z
paper_authors._id: 656f473c14fa8cfccd14559e
paper_authors.avatarUrl: /avatars/8f4fef3d835a7a11c2ab66dbf04f3424.svg
paper_authors.fullname: Lingming Zhang
paper_authors.type: user
paper_authors.user: lingming
paper_authors._id: 65e161d606c9c86f25225a60
paper_authors.name: Sean Hughes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:36:23.095Z
paper_authors._id: 633dd1e467e67df066ef2cbf
paper_authors.avatarUrl: /avatars/121fff6aefd09565fd2b01d7fcda6757.svg
paper_authors.fullname: Sean Hughes
paper_authors.type: user
paper_authors.user: hughesthe1st
paper_authors._id: 65e161d606c9c86f25225a61
paper_authors.name: Thomas Wolf
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:55:17.620Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_authors._id: 65e161d606c9c86f25225a62
paper_authors.name: Arjun Guha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T10:32:06.551Z
paper_authors._id: 62d8315bad693a1a962864b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664332914111-62d8315bad693a1a962864b3.png
paper_authors.fullname: Arjun Guha
paper_authors.type: user
paper_authors.user: arjunguha
paper_authors._id: 65e161d606c9c86f25225a63
paper_authors.name: Leandro von Werra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:39.563Z
paper_authors._id: 5e48005437cb5b49818287a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
paper_authors.fullname: Leandro von Werra
paper_authors.type: user
paper_authors.user: lvwerra
paper_authors._id: 65e161d606c9c86f25225a64
paper_authors.name: Harm de Vries
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:40:58.732Z
paper_authors._id: 62ce91e8b5811ba53c0ce7b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ce91e8b5811ba53c0ce7b9/ZNgd-p0h-q1j374QqEGrb.jpeg
paper_authors.fullname: Harm de Vries
paper_authors.type: user
paper_authors.user: harmdevries
paper_title: StarCoder 2 and The Stack v2: The Next Generation
paper_summary: The BigCode project, an open-scientific collaboration focused on the
  responsible development of Large Language Models for Code (Code LLMs),
  introduces StarCoder2. In partnership with Software Heritage (SWH), we build
  The Stack v2 on top of the digital commons of their source code archive.
  Alongside the SWH repositories spanning 619 programming languages, we carefully
  select other high-quality data sources, such as GitHub pull requests, Kaggle
  notebooks, and code documentation. This results in a training set that is 4x
  larger than the first StarCoder dataset. We train StarCoder2 models with 3B,
  7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate
  them on a comprehensive set of Code LLM benchmarks. We find that our small
  model, StarCoder2-3B, outperforms other Code LLMs of similar size on most
  benchmarks, and also outperforms StarCoderBase-15B. Our large model,
  StarCoder2- 15B, significantly outperforms other models of comparable size. In
  addition, it matches or outperforms CodeLlama-34B, a model more than twice its
  size. Although DeepSeekCoder- 33B is the best-performing model at code
  completion for high-resource languages, we find that StarCoder2-15B outperforms
  it on math and code reasoning benchmarks, as well as several low-resource
  languages. We make the model weights available under an OpenRAIL license and
  ensure full transparency regarding the training data by releasing the SoftWare
  Heritage persistent IDentifiers (SWHIDs) of the source code data.
paper_ai_keywords: StarCoder2 Software Heritage (SWH) Stack v2 Code LLM benchmarks StarCoderBase-15B CodeLlama-34B DeepSeekCoder-33B OpenRAIL license SoftWare Heritage persistent IDentifiers (SWHIDs)