StarCoder 2 and The Stack v2: The Next Generation
The BigCode project, an open-scientific collaboration focused on the
responsible development of Large Language Models for Code (Code LLMs),
introduces StarCoder2. In partnership with Software Heritage (SWH), we build
The Stack v2 on top of the digital commons of their source code archive.
Alongside the SWH repositories spanning 619 programming languages, we carefully
select other high-quality data sources, such as GitHub pull requests, Kaggle
notebooks, and code documentation. This results in a training set that is 4x
larger than the first StarCoder dataset. We train StarCoder2 models with 3B,
7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate
them on a comprehensive set of Code LLM benchmarks. We find that our small
model, StarCoder2-3B, outperforms other Code LLMs of similar size on most
benchmarks, and also outperforms StarCoderBase-15B. Our large model,
StarCoder2- 15B, significantly outperforms other models of comparable size. In
addition, it matches or outperforms CodeLlama-34B, a model more than twice its
size. Although DeepSeekCoder- 33B is the best-performing model at code
completion for high-resource languages, we find that StarCoder2-15B outperforms
it on math and code reasoning benchmarks, as well as several low-resource
languages. We make the model weights available under an OpenRAIL license and
ensure full transparency regarding the training data by releasing the SoftWare
Heritage persistent IDentifiers (SWHIDs) of the source code data.
65e161d606c9c86f25225a23
Anton Lozhkov
admin_assigned
2024-03-01T09:58:06.843Z
602e6dee60e3dd96631c906e
https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png
Anton Lozhkov
user
anton-l
65e161d606c9c86f25225a24
Raymond Li
admin_assigned
2024-03-01T09:58:23.179Z
62d9a30cbfc8699e098791c5
/avatars/c80cf773f436748c992d76ca042dd5fb.svg
Raymond Li
user
RaymondLi
65e161d606c9c86f25225a25
Loubna Ben Allal
claimed_verified
2024-03-01T09:55:19.844Z
61c141342aac764ce1654e43
https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
Loubna Ben Allal
user
loubnabnl
65e161d606c9c86f25225a26
Federico Cassano
admin_assigned
2024-03-01T09:58:30.709Z
642ca13cc3684e5a4e806661
https://cdn-avatars.huggingface.co/v1/production/uploads/642ca13cc3684e5a4e806661/ELhdKsK429zi4wZkVkx9Y.jpeg
Federico Cassano
user
cassanof
65e161d606c9c86f25225a27
Joel Lamy-Poirier
admin_assigned
2024-03-01T09:58:50.211Z
6372ec0e759c1ca82045c43d
/avatars/86d5dbdb10a0cbf9a39a32e2d87361b6.svg
Joel Lamy-Poirier
user
jlamypoirier
65e161d606c9c86f25225a28
Nouamane Tazi
admin_assigned
2024-03-01T09:58:56.576Z
5ff8c9f4b2035d9a81a859f7
https://cdn-avatars.huggingface.co/v1/production/uploads/1652134289581-5ff8c9f4b2035d9a81a859f7.jpeg
Nouamane Tazi
user
nouamanetazi
65e161d606c9c86f25225a29
Ao Tang
admin_assigned
2024-03-01T09:59:15.123Z
645a5024e7e8a4b57bd8e73e
/avatars/6cfbf7138ad554dfdb8052d5edc7d320.svg
Ao Tang
user
thhaus
65e161d606c9c86f25225a2a
Dmytro Pykhtar
admin_assigned
2024-03-01T09:59:35.269Z
62e12c61cb1f164f2cb0afd3
https://cdn-avatars.huggingface.co/v1/production/uploads/1658924005273-noauth.jpeg
Dmytro Pykhtar
user
dpykhtar
65e161d606c9c86f25225a2b
Jiawei Liu
claimed_verified
2024-03-01T09:40:46.334Z
644b584a9279988e0cbeb664
https://cdn-avatars.huggingface.co/v1/production/uploads/644b584a9279988e0cbeb664/fhWCI_Q26tTruhdFkjejw.jpeg
Jiawei Liu
user
ganler
65e161d606c9c86f25225a2c
Yuxiang Wei
admin_assigned
2024-03-01T10:00:10.606Z
632a176259950c1d279d5ea7
https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
Yuxiang Wei
user
yuxiang630
65e161d606c9c86f25225a2d
Tianyang Liu
claimed_verified
2024-03-01T09:26:02.046Z
629e2bcc46b4826be2c57fe3
https://cdn-avatars.huggingface.co/v1/production/uploads/629e2bcc46b4826be2c57fe3/41BiA52XlZi31ABsljFiq.jpeg
Tianyang Liu
user
tianyang
65e161d606c9c86f25225a2e
Max Tian
admin_assigned
2024-03-01T10:00:24.947Z
633fe49ff81b9d10135fe20a
https://cdn-avatars.huggingface.co/v1/production/uploads/633fe49ff81b9d10135fe20a/3s7dJJgF6jnS-DypXXybC.png
Max Tian
user
maxmatical
65e161d606c9c86f25225a2f
Denis Kocetkov
admin_assigned
2024-03-01T10:00:31.401Z
62dea51feee79ca5ef8f834a
/avatars/17e9356625785622762874fe96900599.svg
Denis Kocetkov
user
denisko
65e161d606c9c86f25225a30
Arthur Zucker
admin_assigned
2024-03-01T10:00:44.997Z
62441cb7456803e95009a08f
https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg
Arthur Zucker
user
ArthurZ
65e161d606c9c86f25225a31
Younes Belkada
admin_assigned
2024-03-01T10:00:53.289Z
62441d1d9fdefb55a0b7d12c
https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
Younes B
user
ybelkada
65e161d606c9c86f25225a32
Zijian Wang
admin_assigned
2024-03-01T10:01:21.399Z
6465183934acce85aea2c3b3
/avatars/275a0811381afc980d339ba2aa1294c7.svg
Zijian Wang
user
zijwang
65e161d606c9c86f25225a33
Qian Liu
claimed_verified
2024-03-01T09:40:56.592Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
65e161d606c9c86f25225a34
Dmitry Abulkhanov
admin_assigned
2024-03-01T10:01:30.784Z
636a9a07e3ad78bc68b1a5a2
https://cdn-avatars.huggingface.co/v1/production/uploads/1668020490988-636a9a07e3ad78bc68b1a5a2.jpeg
Dmitry Abulkhanov
user
mponty
65e161d606c9c86f25225a35
Indraneil Paul
admin_assigned
2024-03-01T10:01:36.838Z
62054c0b522e40b4a18d8744
https://cdn-avatars.huggingface.co/v1/production/uploads/62054c0b522e40b4a18d8744/e6bJyJCwEuOZI8BiKC0rx.jpeg
Indraneil Paul
user
iNeil77
65e161d606c9c86f25225a36
Zhuang Li
admin_assigned
2024-03-01T10:02:06.177Z
63d159132036e44c44f87a91
/avatars/6a2c9e5b3b25cf1949277d8c40c0070b.svg
Zhuang Li
user
lizhuang144
65e161d606c9c86f25225a37
Wen-Ding Li
admin_assigned
2024-03-01T10:02:13.737Z
62e221dfcb1f164f2cb8a66b
/avatars/06f05622e232304d3f0b8c291f3263be.svg
Wen-Ding Li
user
xu3kev
65e161d606c9c86f25225a38
Megan Risdal
admin_assigned
2024-03-01T09:40:55.577Z
626b23fc822e3d85324babed
https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
Meg Risdal
user
megrisdal
65e161d606c9c86f25225a39
Jia Li
65e161d606c9c86f25225a3a
Jian Zhu
65e161d606c9c86f25225a3b
Terry Yue Zhuo
claimed_verified
2024-03-01T09:40:44.067Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
65e161d606c9c86f25225a3c
Evgenii Zheltonozhskii
admin_assigned
2024-03-01T10:02:38.479Z
61ae250602b8655e3c8973cd
https://cdn-avatars.huggingface.co/v1/production/uploads/61ae250602b8655e3c8973cd/aXp22-B-oKlm_h6kLx9Uv.jpeg
Evgenii Zheltonozhskii
user
EvgeniyZh
65e161d606c9c86f25225a3d
Nii Osae Osae Dade
claimed_verified
2024-03-02T19:13:44.709Z
621d2948962e5162b2f7d985
/avatars/1a42707be20d6059174a6a43685aa43e.svg
Nii Osae
user
starkosae
65e161d606c9c86f25225a3e
Wenhao Yu
admin_assigned
2024-03-01T10:03:08.512Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
65e161d606c9c86f25225a3f
Lucas Krauß
65e161d606c9c86f25225a40
Naman Jain
claimed_verified
2024-03-02T19:13:56.924Z
6411ed451e42164b9f0f8b24
/avatars/a5c023f407894d3d1dba7b343b847380.svg
Naman Jain
user
StringChaos
65e161d606c9c86f25225a41
Yixuan Su
admin_assigned
2024-03-01T10:04:30.527Z
6144c40da8f4f7baca305243
/avatars/1f9bbb70f859f34563fdd0a7de4cd254.svg
Yixuan Su
user
pangpang666
65e161d606c9c86f25225a42
Xuanli He
admin_assigned
2024-03-01T10:32:16.704Z
614b0ab851b6682b9e56fd92
/avatars/81606ae85106b89a34bd08283646d07b.svg
Xuanli He
user
zodiache
65e161d606c9c86f25225a43
Manan Dey
admin_assigned
2024-03-01T10:32:24.746Z
5fc53305fa6eef7667a4d691
https://cdn-avatars.huggingface.co/v1/production/uploads/1623089209793-5fc53305fa6eef7667a4d691.jpeg
Manan Dey
user
manandey
65e161d606c9c86f25225a44
Edoardo Abati
admin_assigned
2024-03-01T10:32:32.918Z
61be0c0394bd578f72861849
https://cdn-avatars.huggingface.co/v1/production/uploads/1655399022242-61be0c0394bd578f72861849.jpeg
Edoardo Abati
user
EdoAbati
65e161d606c9c86f25225a45
Yekun Chai
admin_assigned
2024-03-04T17:20:25.292Z
626a6ce8adf559c88de9ace0
https://cdn-avatars.huggingface.co/v1/production/uploads/1651142275247-626a6ce8adf559c88de9ace0.jpeg
CYK
user
cyk1337
65e161d606c9c86f25225a46
Niklas Muennighoff
admin_assigned
2024-03-01T10:32:51.477Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
65e161d606c9c86f25225a47
Xiangru Tang
admin_assigned
2024-03-01T10:32:58.810Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
user
RTT1
65e161d606c9c86f25225a48
Muhtasham Oblokulov
claimed_verified
2024-03-01T09:40:50.531Z
5fcaabed246881afd5b00167
https://cdn-avatars.huggingface.co/v1/production/uploads/1651847561574-5fcaabed246881afd5b00167.jpeg
Muhtasham Oblokulov
user
muhtasham
65e161d606c9c86f25225a49
Christopher Akiki
claimed_verified
2024-03-01T09:40:48.537Z
5e70f6048ce3c604d78fe133
https://cdn-avatars.huggingface.co/v1/production/uploads/1646492542174-5e70f6048ce3c604d78fe133.jpeg
Christopher Akiki
user
christopher
65e161d606c9c86f25225a4a
Marc Marone
admin_assigned
2024-03-01T10:33:05.734Z
63e410e88083f19a218be964
/avatars/ce271d1686f6e5ee1f5b2d429cb60de6.svg
Marc Marone
user
mmarone
65e161d606c9c86f25225a4b
Chenghao Mou
admin_assigned
2024-03-01T10:33:14.231Z
607a5b44489fc71534e91c0e
https://cdn-avatars.huggingface.co/v1/production/uploads/1638743128077-607a5b44489fc71534e91c0e.jpeg
Chenghao Mou
user
chenghao
65e161d606c9c86f25225a4c
Mayank Mishra
claimed_verified
2024-03-01T09:40:54.340Z
62cd5057674cdb524450093d
https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
Mayank Mishra
user
mayank-mishra
65e161d606c9c86f25225a4d
Alex Gu
claimed_verified
2024-03-02T19:13:46.853Z
6179abf8ec6ce4dc2e5f2376
/avatars/d10c6a1b350146b36949a24220471295.svg
Alex Gu
user
minimario
65e161d606c9c86f25225a4e
Binyuan Hui
claimed_verified
2024-03-01T09:58:41.147Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
65e161d606c9c86f25225a4f
Tri Dao
admin_assigned
2024-03-01T10:33:39.454Z
64b8a6b5cf14c2fabe98159b
/avatars/dbc009451865435bf290791beadc4723.svg
Tri Dao
user
tridao
65e161d606c9c86f25225a50
Armel Zebaze
65e161d606c9c86f25225a51
Olivier Dehaene
admin_assigned
2024-03-01T10:33:55.167Z
62a093d63e7d1dda047039fc
https://cdn-avatars.huggingface.co/v1/production/uploads/62a093d63e7d1dda047039fc/QGpVSKuJLwl2EsiffCYML.jpeg
Olivier Dehaene
user
olivierdehaene
65e161d606c9c86f25225a52
Nicolas Patry
admin_assigned
2024-03-01T10:34:17.991Z
5e2967b819407e3277369b95
https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png
Nicolas Patry
user
Narsil
65e161d606c9c86f25225a53
Canwen Xu
admin_assigned
2024-03-01T10:34:26.865Z
5e3c29d8f55e2b62848a5224
https://cdn-avatars.huggingface.co/v1/production/uploads/1618318588221-5e3c29d8f55e2b62848a5224.png
Canwen Xu
user
canwenxu
65e161d606c9c86f25225a54
Julian McAuley
65e161d606c9c86f25225a55
Han Hu
65e161d606c9c86f25225a56
Torsten Scholak
admin_assigned
2024-03-01T10:34:47.627Z
60ecaa5efee13fee7ada7af4
https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
Torsten Scholak
user
tscholak
65e161d606c9c86f25225a57
Sebastien Paquet
65e161d606c9c86f25225a58
Jennifer Robinson
admin_assigned
2024-03-01T10:35:09.172Z
6452f03dbeba452314fe59fa
/avatars/ed7e760b93e2a701bfbcf1ecc62ac33c.svg
Jennifer Robinson
user
Jenniferrobinsonservicenow
65e161d606c9c86f25225a59
Carolyn Jane Anderson
admin_assigned
2024-03-01T10:35:19.548Z
6243199444c9c3b21be74c50
https://cdn-avatars.huggingface.co/v1/production/uploads/6243199444c9c3b21be74c50/uxnj03NBcV_TeC3eV2U-Q.jpeg
Carolyn Anderson
user
canders1
65e161d606c9c86f25225a5a
Nicolas Chapados
admin_assigned
2024-03-01T10:35:27.692Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
user
nicolaschapados
65e161d606c9c86f25225a5b
Mostofa Patwary
admin_assigned
2024-03-01T10:35:35.319Z
630544b09d2531fabd156fd3
/avatars/7b5374244a887577834fb4524ff76d01.svg
Mostofa Patwary
user
mpatwary
65e161d606c9c86f25225a5c
Nima Tajbakhsh
admin_assigned
2024-03-01T10:35:42.399Z
6542c90c4f89f533be7609bd
/avatars/9f5ee3709940fe9632c3896be79796a6.svg
Nima Tajbakhsh
user
ntajbakhsh
65e161d606c9c86f25225a5d
Yacine Jernite
admin_assigned
2024-03-01T10:35:50.209Z
5ee3a7cd2a3eae3cbdad1305
https://cdn-avatars.huggingface.co/v1/production/uploads/1594144055859-5ee3a7cd2a3eae3cbdad1305.jpeg
Yacine Jernite
user
yjernite
65e161d606c9c86f25225a5e
Carlos Muñoz Ferrandis
admin_assigned
2024-03-01T10:35:57.932Z
629f05bddaca51e26a7cfef1
https://cdn-avatars.huggingface.co/v1/production/uploads/1654588848873-noauth.jpeg
Carlos Muñoz Ferrandis
user
CarlosMF
65e161d606c9c86f25225a5f
Lingming Zhang
admin_assigned
2024-03-01T10:36:05.659Z
656f473c14fa8cfccd14559e
/avatars/8f4fef3d835a7a11c2ab66dbf04f3424.svg
Lingming Zhang
user
lingming
65e161d606c9c86f25225a60
Sean Hughes
admin_assigned
2024-03-01T10:36:23.095Z
633dd1e467e67df066ef2cbf
/avatars/121fff6aefd09565fd2b01d7fcda6757.svg
Sean Hughes
user
hughesthe1st
65e161d606c9c86f25225a61
Thomas Wolf
claimed_verified
2024-03-01T09:55:17.620Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
65e161d606c9c86f25225a62
Arjun Guha
claimed_verified
2024-03-01T10:32:06.551Z
62d8315bad693a1a962864b3
https://cdn-avatars.huggingface.co/v1/production/uploads/1664332914111-62d8315bad693a1a962864b3.png
Arjun Guha
user
arjunguha
65e161d606c9c86f25225a63
Leandro von Werra
claimed_verified
2024-03-01T09:40:39.563Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
65e161d606c9c86f25225a64
Harm de Vries
claimed_verified
2024-03-01T09:40:58.732Z
62ce91e8b5811ba53c0ce7b9
https://cdn-avatars.huggingface.co/v1/production/uploads/62ce91e8b5811ba53c0ce7b9/ZNgd-p0h-q1j374QqEGrb.jpeg
Harm de Vries
user
harmdevries
StarCoder 2 and The Stack v2: The Next Generation
The BigCode project, an open-scientific collaboration focused on the
responsible development of Large Language Models for Code (Code LLMs),
introduces StarCoder2. In partnership with Software Heritage (SWH), we build
The Stack v2 on top of the digital commons of their source code archive.
Alongside the SWH repositories spanning 619 programming languages, we carefully
select other high-quality data sources, such as GitHub pull requests, Kaggle
notebooks, and code documentation. This results in a training set that is 4x
larger than the first StarCoder dataset. We train StarCoder2 models with 3B,
7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate
them on a comprehensive set of Code LLM benchmarks. We find that our small
model, StarCoder2-3B, outperforms other Code LLMs of similar size on most
benchmarks, and also outperforms StarCoderBase-15B. Our large model,
StarCoder2- 15B, significantly outperforms other models of comparable size. In
addition, it matches or outperforms CodeLlama-34B, a model more than twice its
size. Although DeepSeekCoder- 33B is the best-performing model at code
completion for high-resource languages, we find that StarCoder2-15B outperforms
it on math and code reasoning benchmarks, as well as several low-resource
languages. We make the model weights available under an OpenRAIL license and
ensure full transparency regarding the training data by releasing the SoftWare
Heritage persistent IDentifiers (SWHIDs) of the source code data.
StarCoder2, a large language model for code developed through a collaboration with Software Heritage, outperforms other models of similar size on various benchmarks and matches or outperforms larger models in specific areas.
Large Language Models Code LLMs StarCoder2 Software Heritage The Stack GitHub pull requests Kaggle notebooks code documentation DeepSeekCoder math and code reasoning benchmarks OpenRAIL license SWHIDs