title: CodeChain: Towards Modular Code Generation Through Chain of
    Self-revisions with Representative Sub-modules
summary: Large Language Models (LLMs) have already become quite proficient at solving
  simpler programming tasks like those in HumanEval or MBPP benchmarks. However,
  solving more complex and competitive programming tasks is still quite
  challenging for these models - possibly due to their tendency to generate
  solutions as monolithic code blocks instead of decomposing them into logical
  sub-tasks and sub-modules. On the other hand, experienced programmers
  instinctively write modularized code with abstraction for solving complex
  tasks, often reusing previously developed modules. To address this gap, we
  propose CodeChain, a novel framework for inference that elicits modularized
  code generation through a chain of self-revisions, each being guided by some
  representative sub-modules generated in previous iterations. Concretely,
  CodeChain first instructs the LLM to generate modularized codes through
  chain-of-thought prompting. Then it applies a chain of self-revisions by
  iterating the two steps: 1) extracting and clustering the generated sub-modules
  and selecting the cluster representatives as the more generic and re-usable
  implementations, and 2) augmenting the original chain-of-thought prompt with
  these selected module-implementations and instructing the LLM to re-generate
  new modularized solutions. We find that by naturally encouraging the LLM to
  reuse the previously developed and verified sub-modules, CodeChain can
  significantly boost both modularity as well as correctness of the generated
  solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on
  CodeContests. It is shown to be effective on both OpenAI LLMs as well as
  open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation
  studies with different methods of prompting, number of clusters, model sizes,
  program qualities, etc., to provide useful insights that underpin CodeChain's
  success.
paper_authors._id: 652cc1825aec376f553d4dfe
paper_authors.name: Hung Le
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-19T13:37:11.000Z
paper_authors._id: 63eeee92be5d5900b85367bb
paper_authors.avatarUrl: /avatars/ef10dd6e1ca85861996034797cafe5b8.svg
paper_authors.fullname: Hung Le
paper_authors.type: user
paper_authors.user: hle2023
paper_authors._id: 652cc1825aec376f553d4dff
paper_authors.name: Hailin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:25:05.749Z
paper_authors._id: 646c440036505117e22f5b76
paper_authors.avatarUrl: /avatars/a833947c9d452d673d2598eea7363a73.svg
paper_authors.fullname: Hailin Chen
paper_authors.type: user
paper_authors.user: hailche
paper_authors._id: 652cc1825aec376f553d4e00
paper_authors.name: Amrita Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:25:14.238Z
paper_authors._id: 6461c2905dba83471db3be53
paper_authors.avatarUrl: /avatars/6e36cf86201d590ac729a75d4a439cde.svg
paper_authors.fullname: Amrita Saha
paper_authors.type: user
paper_authors.user: amritasaha87
paper_authors._id: 652cc1825aec376f553d4e01
paper_authors.name: Akash Gokul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:25:21.435Z
paper_authors._id: 63093856b105f8675bd1c783
paper_authors.avatarUrl: /avatars/505a7b99893330a36a136d13fa99e05e.svg
paper_authors.fullname: Akash Gokul
paper_authors.type: user
paper_authors.user: akashgokul
paper_authors._id: 652cc1825aec376f553d4e02
paper_authors.name: Doyen Sahoo
paper_authors._id: 652cc1825aec376f553d4e03
paper_authors.name: Shafiq Joty
paper_title: CodeChain: Towards Modular Code Generation Through Chain of
    Self-revisions with Representative Sub-modules
paper_summary: Large Language Models (LLMs) have already become quite proficient at solving
  simpler programming tasks like those in HumanEval or MBPP benchmarks. However,
  solving more complex and competitive programming tasks is still quite
  challenging for these models - possibly due to their tendency to generate
  solutions as monolithic code blocks instead of decomposing them into logical
  sub-tasks and sub-modules. On the other hand, experienced programmers
  instinctively write modularized code with abstraction for solving complex
  tasks, often reusing previously developed modules. To address this gap, we
  propose CodeChain, a novel framework for inference that elicits modularized
  code generation through a chain of self-revisions, each being guided by some
  representative sub-modules generated in previous iterations. Concretely,
  CodeChain first instructs the LLM to generate modularized codes through
  chain-of-thought prompting. Then it applies a chain of self-revisions by
  iterating the two steps: 1) extracting and clustering the generated sub-modules
  and selecting the cluster representatives as the more generic and re-usable
  implementations, and 2) augmenting the original chain-of-thought prompt with
  these selected module-implementations and instructing the LLM to re-generate
  new modularized solutions. We find that by naturally encouraging the LLM to
  reuse the previously developed and verified sub-modules, CodeChain can
  significantly boost both modularity as well as correctness of the generated
  solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on
  CodeContests. It is shown to be effective on both OpenAI LLMs as well as
  open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation
  studies with different methods of prompting, number of clusters, model sizes,
  program qualities, etc., to provide useful insights that underpin CodeChain's
  success.
paper_ai_summary: CodeChain improves the modularity and correctness of solutions generated by Large Language Models (LLMs) through a chain of self-revisions and sub-module reuse, achieving significant performance gains on competitive programming tasks.
paper_ai_keywords: Large Language Models LLMs HumanEval MBPP modularized code chain-of-thought prompting self-revisions sub-modules abstraction sub-module reuse chain-of-thought CodeChain APPS CodeContests OpenAI WizardCoder ablation studies