title: Which Data Attributes Stimulate Math and Code Reasoning? An
    Investigation via Influence Functions
summary: Large language models (LLMs) have demonstrated remarkable reasoning
  capabilities in math and coding, often bolstered by post-training on the
  chain-of-thoughts (CoTs) generated by stronger models. However, existing
  strategies for curating such training data predominantly rely on heuristics,
  limiting generalizability and failing to capture subtleties underlying in data.
  To address these limitations, we leverage influence functions to systematically
  attribute LLMs' reasoning ability on math and coding to individual training
  examples, sequences, and tokens, enabling deeper insights into effective data
  characteristics. Our Influence-based Reasoning Attribution (Infra) uncovers
  nontrivial cross-domain effects across math and coding tasks: high-difficulty
  math examples improve both math and code reasoning, while low-difficulty code
  tasks most effectively benefit code reasoning. Based on these findings, we
  introduce a simple yet effective dataset reweighting strategy by flipping task
  difficulty, which doubles AIME24 accuracy from 10\% to 20\% and boosts
  LiveCodeBench accuracy from 33.8\% to 35.3\% for Qwen2.5-7B-Instruct. Moreover,
  our fine-grained attribution reveals that the sequence-level exploratory
  behaviors enhance reasoning performance in both math and code, and the
  token-level influence patterns are distinct for math and code reasoning: the
  former prefers natural language logic connectors and the latter emphasizes
  structural syntax.
paper_authors._id: 68352aac38e5ca9eb5349c2f
paper_authors.name: Siqi Kou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:05:20.747Z
paper_authors._id: 654e330f350abceb30a1390b
paper_authors.avatarUrl: /avatars/e54a8be788fa1bdc7acefecc208215bb.svg
paper_authors.fullname: KouSiqi
paper_authors.type: user
paper_authors.user: karrykkk
paper_authors._id: 68352aac38e5ca9eb5349c30
paper_authors.name: Qingyuan Tian
paper_authors._id: 68352aac38e5ca9eb5349c31
paper_authors.name: Hanwen Xu
paper_authors._id: 68352aac38e5ca9eb5349c32
paper_authors.name: Zihao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:44:07.147Z
paper_authors._id: 6544c0585a13979f82038a1c
paper_authors.avatarUrl: /avatars/01f3e862d49020e9eaf1728e4ba97bea.svg
paper_authors.fullname: Zeng Zihao
paper_authors.type: user
paper_authors.user: zzh6666
paper_authors._id: 68352aac38e5ca9eb5349c33
paper_authors.name: Zhijie Deng
paper_title: Which Data Attributes Stimulate Math and Code Reasoning? An
    Investigation via Influence Functions
paper_summary: Large language models (LLMs) have demonstrated remarkable reasoning
  capabilities in math and coding, often bolstered by post-training on the
  chain-of-thoughts (CoTs) generated by stronger models. However, existing
  strategies for curating such training data predominantly rely on heuristics,
  limiting generalizability and failing to capture subtleties underlying in data.
  To address these limitations, we leverage influence functions to systematically
  attribute LLMs' reasoning ability on math and coding to individual training
  examples, sequences, and tokens, enabling deeper insights into effective data
  characteristics. Our Influence-based Reasoning Attribution (Infra) uncovers
  nontrivial cross-domain effects across math and coding tasks: high-difficulty
  math examples improve both math and code reasoning, while low-difficulty code
  tasks most effectively benefit code reasoning. Based on these findings, we
  introduce a simple yet effective dataset reweighting strategy by flipping task
  difficulty, which doubles AIME24 accuracy from 10\% to 20\% and boosts
  LiveCodeBench accuracy from 33.8\% to 35.3\% for Qwen2.5-7B-Instruct. Moreover,
  our fine-grained attribution reveals that the sequence-level exploratory
  behaviors enhance reasoning performance in both math and code, and the
  token-level influence patterns are distinct for math and code reasoning: the
  former prefers natural language logic connectors and the latter emphasizes
  structural syntax.
paper_ai_summary: Influence functions are used to attribute LLMs' reasoning in math and coding to individual training elements, revealing cross-domain effects and enabling a reweighting strategy that improves model accuracy.
paper_ai_keywords: Large language models (LLMs) chain-of-thoughts (CoTs) influence functions attribution data curation reasoning ability high-difficulty math examples low-difficulty code tasks dataset reweighting strategy AIME24 accuracy LiveCodeBench accuracy sequence-level exploratory behaviors token-level influence patterns natural language logic connectors structural syntax parameter-efficient fine-tuning