title: Horizon-Length Prediction: Advancing Fill-in-the-Middle Capabilities for
    Code Generation with Lookahead Planning
summary: Fill-in-the-Middle (FIM) has become integral to code language models,
  enabling generation of missing code given both left and right contexts.
  However, the current FIM training paradigm, which reorders original training
  sequences and then performs regular next-token prediction (NTP), often leads to
  models struggling to generate content that aligns smoothly with the surrounding
  context. Crucially, while existing works rely on rule-based post-processing to
  circumvent this weakness, such methods are not practically usable in
  open-domain code completion tasks as they depend on restrictive,
  dataset-specific assumptions (e.g., generating the same number of lines as in
  the ground truth). Moreover, model performance on FIM tasks deteriorates
  significantly without these unrealistic assumptions.
    We hypothesize that NTP alone is insufficient for models to learn effective
  planning conditioned on the distant right context, a critical factor for
  successful code infilling. To overcome this, we propose Horizon-Length
  Prediction (HLP), a novel training objective that teaches models to predict the
  number of remaining middle tokens (i.e., horizon length) at each step. HLP
  advances FIM with lookahead planning, enabling models to inherently learn
  infilling boundaries for arbitrary left and right contexts without relying on
  dataset-specific post-processing. Our evaluation across different models and
  sizes shows that HLP significantly improves FIM performance by up to 24%
  relatively on diverse benchmarks, across file-level and repository-level, and
  without resorting to unrealistic post-processing methods. Furthermore, the
  enhanced planning capability gained through HLP boosts model performance on
  code reasoning. Importantly, HLP only incurs negligible training overhead and
  no additional inference cost, ensuring its practicality for real-world
  scenarios.
paper_authors._id: 6703dbae126f9ab39faf2c51
paper_authors.name: Yifeng Ding
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-07T13:02:08.380Z
paper_authors._id: 64ff904013f154652687bd80
paper_authors.avatarUrl: /avatars/a04deda1d5c9db0c15a9311806a4b2dc.svg
paper_authors.fullname: Yifeng Ding
paper_authors.type: user
paper_authors.user: YifengDing
paper_authors._id: 6703dbae126f9ab39faf2c52
paper_authors.name: Hantian Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T17:47:12.678Z
paper_authors._id: 637556dafd59a439194d819d
paper_authors.avatarUrl: /avatars/a6241a3adf3bd3b1c3a6ba63933a4117.svg
paper_authors.fullname: Hantian Ding
paper_authors.type: user
paper_authors.user: dhantian
paper_authors._id: 6703dbae126f9ab39faf2c53
paper_authors.name: Shiqi Wang
paper_authors._id: 6703dbae126f9ab39faf2c54
paper_authors.name: Qing Sun
paper_authors._id: 6703dbae126f9ab39faf2c55
paper_authors.name: Varun Kumar
paper_authors._id: 6703dbae126f9ab39faf2c56
paper_authors.name: Zijian Wang
paper_title: Horizon-Length Prediction: Advancing Fill-in-the-Middle Capabilities for
    Code Generation with Lookahead Planning
paper_summary: Fill-in-the-Middle (FIM) has become integral to code language models,
  enabling generation of missing code given both left and right contexts.
  However, the current FIM training paradigm, which reorders original training
  sequences and then performs regular next-token prediction (NTP), often leads to
  models struggling to generate content that aligns smoothly with the surrounding
  context. Crucially, while existing works rely on rule-based post-processing to
  circumvent this weakness, such methods are not practically usable in
  open-domain code completion tasks as they depend on restrictive,
  dataset-specific assumptions (e.g., generating the same number of lines as in
  the ground truth). Moreover, model performance on FIM tasks deteriorates
  significantly without these unrealistic assumptions.
    We hypothesize that NTP alone is insufficient for models to learn effective
  planning conditioned on the distant right context, a critical factor for
  successful code infilling. To overcome this, we propose Horizon-Length
  Prediction (HLP), a novel training objective that teaches models to predict the
  number of remaining middle tokens (i.e., horizon length) at each step. HLP
  advances FIM with lookahead planning, enabling models to inherently learn
  infilling boundaries for arbitrary left and right contexts without relying on
  dataset-specific post-processing. Our evaluation across different models and
  sizes shows that HLP significantly improves FIM performance by up to 24%
  relatively on diverse benchmarks, across file-level and repository-level, and
  without resorting to unrealistic post-processing methods. Furthermore, the
  enhanced planning capability gained through HLP boosts model performance on
  code reasoning. Importantly, HLP only incurs negligible training overhead and
  no additional inference cost, ensuring its practicality for real-world
  scenarios.
paper_ai_keywords: Fill-in-the-Middle (FIM) next-token prediction (NTP) most-of-token prediction lookahead planning horizon length code infilling code reasoning