title: On Leakage of Code Generation Evaluation Datasets
summary: In this paper we consider contamination by code generation test sets, in
  particular in their use in modern large language models. We discuss three
  possible sources of such contamination and show findings supporting each of
  them: (i) direct data leakage, (ii) indirect data leakage through the use of
  synthetic data and (iii) overfitting to evaluation sets during model selection.
    Key to our findings is a new dataset of 161 prompts with their associated
  python solutions, dataset which is released at
  https://huggingface.co/datasets/CohereForAI/lbpp .
paper_authors._id: 668fa3782e309249dbc5d44f
paper_authors.name: Alexandre Matton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:40:09.380Z
paper_authors._id: 645277e7515bd8f741776bac
paper_authors.avatarUrl: /avatars/22e4deac49e0be9f7c121ba7072aeb48.svg
paper_authors.fullname: Alexanadre Matton
paper_authors.type: user
paper_authors.user: alexmatton
paper_authors._id: 668fa3782e309249dbc5d450
paper_authors.name: Tom Sherborne
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:40:15.540Z
paper_authors._id: 5fae7fb3b8423e1d80b8a98f
paper_authors.avatarUrl: /avatars/1c0575701529ae0f15a6fed6834a473c.svg
paper_authors.fullname: Tom Sherborne
paper_authors.type: user
paper_authors.user: tomsherborne
paper_authors._id: 668fa3782e309249dbc5d451
paper_authors.name: Dennis Aumiller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:40:21.598Z
paper_authors._id: 5e844d5992fa4e3c26ddb32a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e844d5992fa4e3c26ddb32a/qnVLOiJchVUxDBd8F-wb_.jpeg
paper_authors.fullname: Dennis Aumiller
paper_authors.type: user
paper_authors.user: dennlinger
paper_authors._id: 668fa3782e309249dbc5d452
paper_authors.name: Elena Tommasone
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:40:27.341Z
paper_authors._id: 668d467c3f80d483b4a55da8
paper_authors.avatarUrl: /avatars/79cb62d6d68baa05cb9c86105cfafa0f.svg
paper_authors.fullname: Elena Tommasone
paper_authors.type: user
paper_authors.user: ElenaT123
paper_authors._id: 668fa3782e309249dbc5d453
paper_authors.name: Milad Alizadeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:40:33.351Z
paper_authors._id: 636ce2671df179e8b843b1f3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636ce2671df179e8b843b1f3/w5aDbG2obh6JFBlY6teBI.jpeg
paper_authors.fullname: Milad Alizadeh
paper_authors.type: user
paper_authors.user: mil-ad
paper_authors._id: 668fa3782e309249dbc5d454
paper_authors.name: Jingyi He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:40:39.751Z
paper_authors._id: 663ca12d5eee63f75dbf9f24
paper_authors.avatarUrl: /avatars/afec31c33ffe3387ede96a8481908d6f.svg
paper_authors.fullname: Jingyi He
paper_authors.type: user
paper_authors.user: Hollysshelter
paper_authors._id: 668fa3782e309249dbc5d455
paper_authors.name: Raymond Ma
paper_authors._id: 668fa3782e309249dbc5d456
paper_authors.name: Maxime Voisin
paper_authors._id: 668fa3782e309249dbc5d457
paper_authors.name: Ellen Gilsenan-McMahon
paper_authors._id: 668fa3782e309249dbc5d458
paper_authors.name: Matthias Gallé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:41:27.369Z
paper_authors._id: 607fc2824ad99100d63ce35f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634907979945-607fc2824ad99100d63ce35f.png
paper_authors.fullname: Matthias Gallé
paper_authors.type: user
paper_authors.user: mgalle
paper_title: On Leakage of Code Generation Evaluation Datasets
paper_summary: In this paper we consider contamination by code generation test sets, in
  particular in their use in modern large language models. We discuss three
  possible sources of such contamination and show findings supporting each of
  them: (i) direct data leakage, (ii) indirect data leakage through the use of
  synthetic data and (iii) overfitting to evaluation sets during model selection.
    Key to our findings is a new dataset of 161 prompts with their associated
  python solutions, dataset which is released at
  https://huggingface.co/datasets/CohereForAI/lbpp .
paper_ai_keywords: code generation large language models data leakage synthetic data overfitting evaluation sets model selection prompts python solutions