title: AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong
    Pretraining Data Selection
summary: Recently, there has been growing interest in collecting reasoning-intensive
  pretraining data to improve LLMs' complex reasoning ability. Prior approaches
  typically rely on supervised classifiers to identify such data, which requires
  labeling by humans or LLMs, often introducing domain-specific biases. Due to
  the attention heads being crucial to in-context reasoning, we propose
  AttentionInfluence, a simple yet effective, training-free method without
  supervision signal. Our approach enables a small pretrained language model to
  act as a strong data selector through a simple attention head masking
  operation. Specifically, we identify retrieval heads and compute the loss
  difference when masking these heads. We apply AttentionInfluence to a
  1.3B-parameter dense model to conduct data selection on the SmolLM corpus of
  241B tokens, and mix the SmolLM corpus with the selected subset comprising 73B
  tokens to pretrain a 7B-parameter dense model using 1T training tokens and WSD
  learning rate scheduling. Our experimental results demonstrate substantial
  improvements, ranging from 1.4pp to 3.5pp, across several knowledge-intensive
  and reasoning-heavy benchmarks (i.e., MMLU, MMLU-Pro, AGIEval-en, GSM8K, and
  HumanEval). This demonstrates an effective weak-to-strong scaling property,
  with small models improving the final performance of larger models-offering a
  promising and scalable path for reasoning-centric data selection.
paper_authors._id: 6822d1f9c9cb5b62cd606495
paper_authors.name: Kai Hua
paper_authors._id: 6822d1f9c9cb5b62cd606496
paper_authors.name: Steven Wu
paper_authors._id: 6822d1f9c9cb5b62cd606497
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:11:29.702Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6822d1f9c9cb5b62cd606498
paper_authors.name: Ke Shen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-24T18:23:35.265Z
paper_authors._id: 645604eebabbbbd3486dc615
paper_authors.avatarUrl: /avatars/17a5ca8274e2bfc8f183a4af9878a930.svg
paper_authors.fullname: shenke
paper_authors.type: user
paper_authors.user: shenke18
paper_title: AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong
    Pretraining Data Selection
paper_summary: Recently, there has been growing interest in collecting reasoning-intensive
  pretraining data to improve LLMs' complex reasoning ability. Prior approaches
  typically rely on supervised classifiers to identify such data, which requires
  labeling by humans or LLMs, often introducing domain-specific biases. Due to
  the attention heads being crucial to in-context reasoning, we propose
  AttentionInfluence, a simple yet effective, training-free method without
  supervision signal. Our approach enables a small pretrained language model to
  act as a strong data selector through a simple attention head masking
  operation. Specifically, we identify retrieval heads and compute the loss
  difference when masking these heads. We apply AttentionInfluence to a
  1.3B-parameter dense model to conduct data selection on the SmolLM corpus of
  241B tokens, and mix the SmolLM corpus with the selected subset comprising 73B
  tokens to pretrain a 7B-parameter dense model using 1T training tokens and WSD
  learning rate scheduling. Our experimental results demonstrate substantial
  improvements, ranging from 1.4pp to 3.5pp, across several knowledge-intensive
  and reasoning-heavy benchmarks (i.e., MMLU, MMLU-Pro, AGIEval-en, GSM8K, and
  HumanEval). This demonstrates an effective weak-to-strong scaling property,
  with small models improving the final performance of larger models-offering a
  promising and scalable path for reasoning-centric data selection.
paper_ai_summary: AttentionInfluence, a training-free, unsupervised method using attention head masking, improves data selection for pretraining LLMs, enhancing their reasoning abilities across various benchmarks.
paper_ai_keywords: LLMs complex reasoning ability supervised classifiers domain-specific biases attention heads in-context reasoning AttentionInfluence attention head masking retrieval heads SmolLM corpus pretraining 7B-parameter dense model WSD learning rate scheduling MMLU MMLU-Pro AGIEval-en GSM8K HumanEval weak-to-strong scaling property