title: The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating
    Large Language Models
summary: The recent rise in the popularity of large language models has spurred the
  development of extensive code datasets needed to train them. This has left
  limited code available for collection and use in the downstream investigation
  of specific behaviors, or evaluation of large language models without suffering
  from data contamination. To address this problem, we release The Heap, a large
  multilingual dataset covering 57 programming languages that has been
  deduplicated with respect to other open datasets of code, enabling researchers
  to conduct fair evaluations of large language models without significant data
  cleaning overhead.
paper_authors._id: 678a632dd03f325c5a5ad954
paper_authors.name: Jonathan Katzy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T17:05:55.731Z
paper_authors._id: 643d13b04816d7cb420966dc
paper_authors.avatarUrl: /avatars/4f4875e35fa57c2b958cdb5dd6b70c27.svg
paper_authors.fullname: Jonathan katzy
paper_authors.type: user
paper_authors.user: Jkatzy
paper_authors._id: 678a632dd03f325c5a5ad955
paper_authors.name: Razvan Mihai Popescu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T17:05:53.530Z
paper_authors._id: 6446fea4bd9e0e0edf521d49
paper_authors.avatarUrl: /avatars/493c16b52e1a9a940b4e02b6cd6c6cc3.svg
paper_authors.fullname: Razvan Popescu
paper_authors.type: user
paper_authors.user: Razvan27
paper_authors._id: 678a632dd03f325c5a5ad956
paper_authors.name: Arie van Deursen
paper_authors._id: 678a632dd03f325c5a5ad957
paper_authors.name: Maliheh Izadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-17T14:25:21.390Z
paper_authors._id: 63dd4b2af37111482526e0e9
paper_authors.avatarUrl: /avatars/ffd2a07c61ea6272009df6184cb9dcef.svg
paper_authors.fullname: Maliheh Izadi
paper_authors.type: user
paper_authors.user: MalihehIzadi
paper_title: The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating
    Large Language Models
paper_summary: The recent rise in the popularity of large language models has spurred the
  development of extensive code datasets needed to train them. This has left
  limited code available for collection and use in the downstream investigation
  of specific behaviors, or evaluation of large language models without suffering
  from data contamination. To address this problem, we release The Heap, a large
  multilingual dataset covering 57 programming languages that has been
  deduplicated with respect to other open datasets of code, enabling researchers
  to conduct fair evaluations of large language models without significant data
  cleaning overhead.
paper_ai_summary: A large multilingual code dataset, The Heap, enables fair evaluations of language models by minimizing data contamination and cleaning overhead.
paper_ai_keywords: large language models code datasets deduplicated multilingual programming languages fair evaluations data contamination data cleaning