title: Case2Code: Learning Inductive Reasoning with Synthetic Data
summary: Complex reasoning is an impressive ability shown by large language models
  (LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought
  prompting or iterative tool-using to solve challenging tasks step-by-step. In
  this paper, we hope to focus on evaluating and teaching LLMs to conduct
  inductive reasoning, that is, LLMs are supposed to infer underlying rules by
  observing examples or sequential transformations. However, collecting
  large-scale and diverse human-generated inductive data is challenging. We focus
  on data synthesis in the code domain and propose a Case2Code task by
  exploiting the expressiveness and correctness of programs. Specifically, we
  collect a diverse set of executable programs, synthesize input-output
  transformations for each program, and force LLMs to infer the underlying code
  implementations based on the synthetic I/O cases. We first evaluate
  representative LLMs on the synthesized Case2Code task and demonstrate that the
  Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale
  Case2Code training samples to train LLMs to perform inductive reasoning.
  Experimental results show that such induction training benefits not only in
  distribution Case2Code performance but also enhances various coding abilities
  of trained LLMs, demonstrating the great potential of learning inductive
  reasoning via synthetic data.
paper_authors._id: 669895f5691314b30856c64c
paper_authors.name: Yunfan Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:20.418Z
paper_authors._id: 613b08710fcccdd3b430478a
paper_authors.avatarUrl: /avatars/6bb368110cd50996b26046cd09ab9bcc.svg
paper_authors.fullname: Yunfan Shao
paper_authors.type: user
paper_authors.user: yf
paper_authors._id: 669895f5691314b30856c64d
paper_authors.name: Linyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:16.026Z
paper_authors._id: 6698ad87a9778d20ff373142
paper_authors.avatarUrl: /avatars/5e3546d124e8c4f4636a41eab6f0cf68.svg
paper_authors.fullname: linyangli
paper_authors.type: user
paper_authors.user: linyangli
paper_authors._id: 669895f5691314b30856c64e
paper_authors.name: Yichuan Ma
paper_authors._id: 669895f5691314b30856c64f
paper_authors.name: Peiji Li
paper_authors._id: 669895f5691314b30856c650
paper_authors.name: Demin Song
paper_authors._id: 669895f5691314b30856c651
paper_authors.name: Qinyuan Cheng
paper_authors._id: 669895f5691314b30856c652
paper_authors.name: Shimin Li
paper_authors._id: 669895f5691314b30856c653
paper_authors.name: Xiaonan Li
paper_authors._id: 669895f5691314b30856c654
paper_authors.name: Pengyu Wang
paper_authors._id: 669895f5691314b30856c655
paper_authors.name: Qipeng Guo
paper_authors._id: 669895f5691314b30856c656
paper_authors.name: Hang Yan
paper_authors._id: 669895f5691314b30856c657
paper_authors.name: Xipeng Qiu
paper_authors._id: 669895f5691314b30856c658
paper_authors.name: Xuanjing Huang
paper_authors._id: 669895f5691314b30856c659
paper_authors.name: Dahua Lin
paper_title: Case2Code: Learning Inductive Reasoning with Synthetic Data
paper_summary: Complex reasoning is an impressive ability shown by large language models
  (LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought
  prompting or iterative tool-using to solve challenging tasks step-by-step. In
  this paper, we hope to focus on evaluating and teaching LLMs to conduct
  inductive reasoning, that is, LLMs are supposed to infer underlying rules by
  observing examples or sequential transformations. However, collecting
  large-scale and diverse human-generated inductive data is challenging. We focus
  on data synthesis in the code domain and propose a Case2Code task by
  exploiting the expressiveness and correctness of programs. Specifically, we
  collect a diverse set of executable programs, synthesize input-output
  transformations for each program, and force LLMs to infer the underlying code
  implementations based on the synthetic I/O cases. We first evaluate
  representative LLMs on the synthesized Case2Code task and demonstrate that the
  Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale
  Case2Code training samples to train LLMs to perform inductive reasoning.
  Experimental results show that such induction training benefits not only in
  distribution Case2Code performance but also enhances various coding abilities
  of trained LLMs, demonstrating the great potential of learning inductive
  reasoning via synthetic data.
paper_ai_summary: Training LLMs with synthetic data improves their inductive reasoning and coding abilities by enabling them to infer underlying code implementations from input-output transformations.
paper_ai_keywords: LLMs deductive reasoning chain-of-thought prompting iterative tool-using inductive reasoning human-generated inductive data data synthesis code domain Case2Code task executable programs input-output transformations Case-to-code induction inductive training