On Leakage of Code Generation Evaluation Datasets
In this paper we consider contamination by code generation test sets, in
particular in their use in modern large language models. We discuss three
possible sources of such contamination and show findings supporting each of
them: (i) direct data leakage, (ii) indirect data leakage through the use of
synthetic data and (iii) overfitting to evaluation sets during model selection.
  Key to our findings is a new dataset of 161 prompts with their associated
python solutions, dataset which is released at
https://huggingface.co/datasets/CohereForAI/lbpp .
668fa3782e309249dbc5d44f
Alexandre Matton
admin_assigned
2024-07-11T10:40:09.380Z
645277e7515bd8f741776bac
/avatars/22e4deac49e0be9f7c121ba7072aeb48.svg
Alexanadre Matton
user
alexmatton
668fa3782e309249dbc5d450
Tom Sherborne
admin_assigned
2024-07-11T10:40:15.540Z
5fae7fb3b8423e1d80b8a98f
/avatars/1c0575701529ae0f15a6fed6834a473c.svg
Tom Sherborne
user
tomsherborne
668fa3782e309249dbc5d451
Dennis Aumiller
admin_assigned
2024-07-11T10:40:21.598Z
5e844d5992fa4e3c26ddb32a
https://cdn-avatars.huggingface.co/v1/production/uploads/5e844d5992fa4e3c26ddb32a/qnVLOiJchVUxDBd8F-wb_.jpeg
Dennis Aumiller
user
dennlinger
668fa3782e309249dbc5d452
Elena Tommasone
admin_assigned
2024-07-11T10:40:27.341Z
668d467c3f80d483b4a55da8
/avatars/79cb62d6d68baa05cb9c86105cfafa0f.svg
Elena Tommasone
user
ElenaT123
668fa3782e309249dbc5d453
Milad Alizadeh
admin_assigned
2024-07-11T10:40:33.351Z
636ce2671df179e8b843b1f3
https://cdn-avatars.huggingface.co/v1/production/uploads/636ce2671df179e8b843b1f3/w5aDbG2obh6JFBlY6teBI.jpeg
Milad Alizadeh
user
mil-ad
668fa3782e309249dbc5d454
Jingyi He
admin_assigned
2024-07-11T10:40:39.751Z
663ca12d5eee63f75dbf9f24
/avatars/afec31c33ffe3387ede96a8481908d6f.svg
Jingyi He
user
Hollysshelter
668fa3782e309249dbc5d455
Raymond Ma
668fa3782e309249dbc5d456
Maxime Voisin
668fa3782e309249dbc5d457
Ellen Gilsenan-McMahon
668fa3782e309249dbc5d458
Matthias Gallé
admin_assigned
2024-07-11T10:41:27.369Z
607fc2824ad99100d63ce35f
https://cdn-avatars.huggingface.co/v1/production/uploads/1634907979945-607fc2824ad99100d63ce35f.png
Matthias Gallé
user
mgalle
On Leakage of Code Generation Evaluation Datasets
In this paper we consider contamination by code generation test sets, in
particular in their use in modern large language models. We discuss three
possible sources of such contamination and show findings supporting each of
them: (i) direct data leakage, (ii) indirect data leakage through the use of
synthetic data and (iii) overfitting to evaluation sets during model selection.
  Key to our findings is a new dataset of 161 prompts with their associated
python solutions, dataset which is released at
https://huggingface.co/datasets/CohereForAI/lbpp .
Modern large language models exhibit contamination from code generation test sets due to direct data leakage, indirect data leakage through synthetic data, and overfitting to evaluation sets during model selection.
code generation large language models data leakage synthetic data overfitting evaluation sets