title: RepoFusion: Training Code Models to Understand Your Repository
summary: Despite the huge success of Large Language Models (LLMs) in coding assistants
  like GitHub Copilot, these models struggle to understand the context present in
  the repository (e.g., imports, parent classes, files with similar names, etc.),
  thereby producing inaccurate code completions. This effect is more pronounced
  when using these assistants for repositories that the model has not seen during
  training, such as proprietary software or work-in-progress code projects.
  Recent work has shown the promise of using context from the repository during
  inference. In this work, we extend this idea and propose RepoFusion, a
  framework to train models to incorporate relevant repository context.
  Experiments on single-line code completion show that our models trained with
  repository context significantly outperform much larger code models as
  CodeGen-16B-multi (sim73times larger) and closely match the performance of
  the sim 70times larger StarCoderBase model that was trained with the
  Fill-in-the-Middle objective. We find these results to be a novel and
  compelling demonstration of the gains that training with repository context can
  bring. We carry out extensive ablation studies to investigate the impact of
  design choices such as context type, number of contexts, context length, and
  initialization within our framework. Lastly, we release Stack-Repo, a dataset
  of 200 Java repositories with permissive licenses and near-deduplicated files
  that are augmented with three types of repository contexts. Additionally, we
  are making available the code and trained checkpoints for our work. Our
  released resources can be found at https://huggingface.co/RepoFusion.
paper_authors._id: 649268cecd6ab684bd81183a
paper_authors.name: Disha Shrivastava
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:40:34.994Z
paper_authors._id: 6438284b819f3ab20d185a3b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6438284b819f3ab20d185a3b/b29SoH7_Bji6GfCszjqOE.jpeg
paper_authors.fullname: Disha Shrivastava
paper_authors.type: user
paper_authors.user: shrivasd
paper_authors._id: 649268cecd6ab684bd81183b
paper_authors.name: Denis Kocetkov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:41:09.501Z
paper_authors._id: 62dea51feee79ca5ef8f834a
paper_authors.avatarUrl: /avatars/17e9356625785622762874fe96900599.svg
paper_authors.fullname: Denis Kocetkov
paper_authors.type: user
paper_authors.user: denisko
paper_authors._id: 649268cecd6ab684bd81183c
paper_authors.name: Harm de Vries
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:41:33.325Z
paper_authors._id: 62ce91e8b5811ba53c0ce7b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ce91e8b5811ba53c0ce7b9/ZNgd-p0h-q1j374QqEGrb.jpeg
paper_authors.fullname: Harm de Vries
paper_authors.type: user
paper_authors.user: harmdevries
paper_authors._id: 649268cecd6ab684bd81183d
paper_authors.name: Dzmitry Bahdanau
paper_authors._id: 649268cecd6ab684bd81183e
paper_authors.name: Torsten Scholak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:43:04.731Z
paper_authors._id: 60ecaa5efee13fee7ada7af4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
paper_authors.fullname: Torsten Scholak
paper_authors.type: user
paper_authors.user: tscholak
paper_title: RepoFusion: Training Code Models to Understand Your Repository
paper_summary: Despite the huge success of Large Language Models (LLMs) in coding assistants
  like GitHub Copilot, these models struggle to understand the context present in
  the repository (e.g., imports, parent classes, files with similar names, etc.),
  thereby producing inaccurate code completions. This effect is more pronounced
  when using these assistants for repositories that the model has not seen during
  training, such as proprietary software or work-in-progress code projects.
  Recent work has shown the promise of using context from the repository during
  inference. In this work, we extend this idea and propose RepoFusion, a
  framework to train models to incorporate relevant repository context.
  Experiments on single-line code completion show that our models trained with
  repository context significantly outperform much larger code models as
  CodeGen-16B-multi (sim73times larger) and closely match the performance of
  the sim 70times larger StarCoderBase model that was trained with the
  Fill-in-the-Middle objective. We find these results to be a novel and
  compelling demonstration of the gains that training with repository context can
  bring. We carry out extensive ablation studies to investigate the impact of
  design choices such as context type, number of contexts, context length, and
  initialization within our framework. Lastly, we release Stack-Repo, a dataset
  of 200 Java repositories with permissive licenses and near-deduplicated files
  that are augmented with three types of repository contexts. Additionally, we
  are making available the code and trained checkpoints for our work. Our
  released resources can be found at https://huggingface.co/RepoFusion.
paper_ai_keywords: Large Language Models (LLMs) GitHub Copilot repository context denoising models code completions CodeGen-16B-multi StarCoderBase Fill-in-the-Middle objective RepoFusion Stack-Repo context type context length initialization