Infinity Instruct: Scaling Instruction Selection and Synthesis to
  Enhance Language Models
Large Language Models (LLMs) demonstrate strong performance in real-world
applications, yet existing open-source instruction datasets often concentrate
on narrow domains, such as mathematics or coding, limiting generalization and
widening the gap with proprietary models. To bridge this gap, we introduce
Infinity-Instruct, a high-quality instruction dataset designed to enhance both
foundational and chat capabilities of LLMs through a two-phase pipeline. In
Phase 1, we curate 7.4M high-quality foundational instructions
(InfInstruct-F-7.4M) from over 100M samples using hybrid data selection
techniques. In Phase 2, we synthesize 1.5M high-quality chat instructions
(InfInstruct-G-1.5M) through a two-stage process involving instruction
selection, evolution, and diagnostic filtering. We empirically evaluate
Infinity-Instruct by fine-tuning several open-source models, including Mistral,
LLaMA, Qwen, and Yi, and observe substantial performance gains across both
foundational and instruction following benchmarks, consistently surpassing
official instruction-tuned counterparts. Notably, InfInstruct-LLaMA3.1-70B
outperforms GPT-4-0314 by 8.6\% on instruction following tasks while achieving
comparable foundational performance. These results underscore the synergy
between foundational and chat training and offer new insights into holistic LLM
development. Our
datasethttps://huggingface.co/datasets/BAAI/Infinity-Instruct and
codeshttps://gitee.com/li-touch/infinity-instruct have been publicly
released.
685021682932e11c891b582e
Jijie Li
685021682932e11c891b582f
Li Du
685021682932e11c891b5830
Hanyu Zhao
685021682932e11c891b5831
Bo-wen Zhang
685021682932e11c891b5832
Liangdong Wang
claimed_verified
2025-06-16T15:50:45.345Z
63a11ce02fabbbb899a01d58
/avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
ldwang
user
ldwang
685021682932e11c891b5833
Boyan Gao
685021682932e11c891b5834
Guang Liu
claimed_verified
2025-06-16T13:58:12.267Z
632c234f42c386ebd2710434
https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
Guang Liu
user
ZacLiu
685021682932e11c891b5835
Yonghua Lin
Infinity Instruct: Scaling Instruction Selection and Synthesis to
  Enhance Language Models
Large Language Models (LLMs) demonstrate strong performance in real-world
applications, yet existing open-source instruction datasets often concentrate
on narrow domains, such as mathematics or coding, limiting generalization and
widening the gap with proprietary models. To bridge this gap, we introduce
Infinity-Instruct, a high-quality instruction dataset designed to enhance both
foundational and chat capabilities of LLMs through a two-phase pipeline. In
Phase 1, we curate 7.4M high-quality foundational instructions
(InfInstruct-F-7.4M) from over 100M samples using hybrid data selection
techniques. In Phase 2, we synthesize 1.5M high-quality chat instructions
(InfInstruct-G-1.5M) through a two-stage process involving instruction
selection, evolution, and diagnostic filtering. We empirically evaluate
Infinity-Instruct by fine-tuning several open-source models, including Mistral,
LLaMA, Qwen, and Yi, and observe substantial performance gains across both
foundational and instruction following benchmarks, consistently surpassing
official instruction-tuned counterparts. Notably, InfInstruct-LLaMA3.1-70B
outperforms GPT-4-0314 by 8.6\% on instruction following tasks while achieving
comparable foundational performance. These results underscore the synergy
between foundational and chat training and offer new insights into holistic LLM
development. Our
datasethttps://huggingface.co/datasets/BAAI/Infinity-Instruct and
codeshttps://gitee.com/li-touch/infinity-instruct have been publicly
released.
Infinity-Instruct, a comprehensive instruction dataset, enhances both foundational and chat capabilities of large language models through curation and synthesis, achieving superior performance compared to existing datasets.
large language models instruction datasets foundational instructions chat instructions hybrid data selection instruction selection evolution diagnostic filtering fine-tuning instruction following benchmarks