Case2Code: Learning Inductive Reasoning with Synthetic Data
Complex reasoning is an impressive ability shown by large language models
(LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought
prompting or iterative tool-using to solve challenging tasks step-by-step. In
this paper, we hope to focus on evaluating and teaching LLMs to conduct
inductive reasoning, that is, LLMs are supposed to infer underlying rules by
observing examples or sequential transformations. However, collecting
large-scale and diverse human-generated inductive data is challenging. We focus
on data synthesis in the code domain and propose a Case2Code task by
exploiting the expressiveness and correctness of programs. Specifically, we
collect a diverse set of executable programs, synthesize input-output
transformations for each program, and force LLMs to infer the underlying code
implementations based on the synthetic I/O cases. We first evaluate
representative LLMs on the synthesized Case2Code task and demonstrate that the
Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale
Case2Code training samples to train LLMs to perform inductive reasoning.
Experimental results show that such induction training benefits not only in
distribution Case2Code performance but also enhances various coding abilities
of trained LLMs, demonstrating the great potential of learning inductive
reasoning via synthetic data.
669895f5691314b30856c64c
Yunfan Shao
claimed_verified
2024-07-18T09:05:20.418Z
613b08710fcccdd3b430478a
/avatars/6bb368110cd50996b26046cd09ab9bcc.svg
Yunfan Shao
user
yf
669895f5691314b30856c64d
Linyang Li
claimed_verified
2024-07-18T09:05:16.026Z
6698ad87a9778d20ff373142
/avatars/5e3546d124e8c4f4636a41eab6f0cf68.svg
linyangli
user
linyangli
669895f5691314b30856c64e
Yichuan Ma
669895f5691314b30856c64f
Peiji Li
669895f5691314b30856c650
Demin Song
669895f5691314b30856c651
Qinyuan Cheng
claimed_verified
2025-06-24T08:15:33.717Z
63ec4715c81b6a52391c46b8
/avatars/496819b5075a1a834a2b9edeb068c80e.svg
QinyuanCheng
user
Cqy2019
669895f5691314b30856c652
Shimin Li
669895f5691314b30856c653
Xiaonan Li
669895f5691314b30856c654
Pengyu Wang
669895f5691314b30856c655
Qipeng Guo
669895f5691314b30856c656
Hang Yan
669895f5691314b30856c657
Xipeng Qiu
669895f5691314b30856c658
Xuanjing Huang
669895f5691314b30856c659
Dahua Lin
Case2Code: Learning Inductive Reasoning with Synthetic Data
Complex reasoning is an impressive ability shown by large language models
(LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought
prompting or iterative tool-using to solve challenging tasks step-by-step. In
this paper, we hope to focus on evaluating and teaching LLMs to conduct
inductive reasoning, that is, LLMs are supposed to infer underlying rules by
observing examples or sequential transformations. However, collecting
large-scale and diverse human-generated inductive data is challenging. We focus
on data synthesis in the code domain and propose a Case2Code task by
exploiting the expressiveness and correctness of programs. Specifically, we
collect a diverse set of executable programs, synthesize input-output
transformations for each program, and force LLMs to infer the underlying code
implementations based on the synthetic I/O cases. We first evaluate
representative LLMs on the synthesized Case2Code task and demonstrate that the
Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale
Case2Code training samples to train LLMs to perform inductive reasoning.
Experimental results show that such induction training benefits not only in
distribution Case2Code performance but also enhances various coding abilities
of trained LLMs, demonstrating the great potential of learning inductive
reasoning via synthetic data.
Training LLMs with synthetic data improves their inductive reasoning and coding abilities by enabling them to infer underlying code implementations from input-output transformations.
LLMs deductive reasoning chain-of-thought prompting iterative tool-using inductive reasoning human-generated inductive data data synthesis code domain Case2Code task executable programs input-output transformations Case-to-code induction inductive training