AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong
  Pretraining Data Selection
Recently, there has been growing interest in collecting reasoning-intensive
pretraining data to improve LLMs' complex reasoning ability. Prior approaches
typically rely on supervised classifiers to identify such data, which requires
labeling by humans or LLMs, often introducing domain-specific biases. Due to
the attention heads being crucial to in-context reasoning, we propose
AttentionInfluence, a simple yet effective, training-free method without
supervision signal. Our approach enables a small pretrained language model to
act as a strong data selector through a simple attention head masking
operation. Specifically, we identify retrieval heads and compute the loss
difference when masking these heads. We apply AttentionInfluence to a
1.3B-parameter dense model to conduct data selection on the SmolLM corpus of
241B tokens, and mix the SmolLM corpus with the selected subset comprising 73B
tokens to pretrain a 7B-parameter dense model using 1T training tokens and WSD
learning rate scheduling. Our experimental results demonstrate substantial
improvements, ranging from 1.4pp to 3.5pp, across several knowledge-intensive
and reasoning-heavy benchmarks (i.e., MMLU, MMLU-Pro, AGIEval-en, GSM8K, and
HumanEval). This demonstrates an effective weak-to-strong scaling property,
with small models improving the final performance of larger models-offering a
promising and scalable path for reasoning-centric data selection.
6822d1f9c9cb5b62cd606495
Kai Hua
6822d1f9c9cb5b62cd606496
Steven Wu
6822d1f9c9cb5b62cd606497
Ge Zhang
claimed_verified
2025-05-13T08:11:29.702Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6822d1f9c9cb5b62cd606498
Ke Shen
extracted_confirmed
2025-05-24T18:23:35.265Z
645604eebabbbbd3486dc615
/avatars/17a5ca8274e2bfc8f183a4af9878a930.svg
shenke
user
shenke18
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong
  Pretraining Data Selection
Recently, there has been growing interest in collecting reasoning-intensive
pretraining data to improve LLMs' complex reasoning ability. Prior approaches
typically rely on supervised classifiers to identify such data, which requires
labeling by humans or LLMs, often introducing domain-specific biases. Due to
the attention heads being crucial to in-context reasoning, we propose
AttentionInfluence, a simple yet effective, training-free method without
supervision signal. Our approach enables a small pretrained language model to
act as a strong data selector through a simple attention head masking
operation. Specifically, we identify retrieval heads and compute the loss
difference when masking these heads. We apply AttentionInfluence to a
1.3B-parameter dense model to conduct data selection on the SmolLM corpus of
241B tokens, and mix the SmolLM corpus with the selected subset comprising 73B
tokens to pretrain a 7B-parameter dense model using 1T training tokens and WSD
learning rate scheduling. Our experimental results demonstrate substantial
improvements, ranging from 1.4pp to 3.5pp, across several knowledge-intensive
and reasoning-heavy benchmarks (i.e., MMLU, MMLU-Pro, AGIEval-en, GSM8K, and
HumanEval). This demonstrates an effective weak-to-strong scaling property,
with small models improving the final performance of larger models-offering a
promising and scalable path for reasoning-centric data selection.
AttentionInfluence, a training-free, unsupervised method using attention head masking, improves data selection for pretraining LLMs, enhancing their reasoning abilities across various benchmarks.
LLMs complex reasoning ability supervised classifiers domain-specific biases attention heads in-context reasoning AttentionInfluence attention head masking retrieval heads SmolLM corpus pretraining 7B-parameter dense model WSD learning rate scheduling MMLU MMLU-Pro AGIEval-en GSM8K HumanEval weak-to-strong scaling property