title: Rethinking Data Selection at Scale: Random Selection is Almost All You
    Need
summary: Supervised fine-tuning (SFT) is crucial for aligning Large Language Models
  (LLMs) with human instructions. The primary goal during SFT is to select a
  small yet representative subset of training data from the larger pool, such
  that fine-tuning with this subset achieves results comparable to or even
  exceeding those obtained using the entire dataset. However, most existing data
  selection techniques are designed for small-scale data pools, which fail to
  meet the demands of real-world SFT scenarios. In this paper, we replicated
  several self-scoring methods those that do not rely on external model
  assistance on two million scale datasets, and found that nearly all methods
  struggled to significantly outperform random selection when dealing with such
  large-scale data pools. Moreover, our comparisons suggest that, during SFT,
  diversity in data selection is more critical than simply focusing on high
  quality data. We also analyzed the limitations of several current approaches,
  explaining why they perform poorly on large-scale datasets and why they are
  unsuitable for such contexts. Finally, we found that filtering data by token
  length offers a stable and efficient method for improving results. This
  approach, particularly when training on long text data, proves highly
  beneficial for relatively weaker base models, such as Llama3.
paper_authors._id: 670dd0cc487a5e24b27713b9
paper_authors.name: Tingyu Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:22.078Z
paper_authors._id: 653b5cb0d994e992e2831539
paper_authors.avatarUrl: /avatars/e6b917cb263f10a6d9c72ec539276428.svg
paper_authors.fullname: xiatingyu
paper_authors.type: user
paper_authors.user: xiatingyu
paper_authors._id: 670dd0cc487a5e24b27713ba
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:21:17.898Z
paper_authors._id: 6438b43ab2ea24b52ebac2b9
paper_authors.avatarUrl: /avatars/84133cd719a4b1e2f5c1a74178425f86.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: bwy
paper_authors._id: 670dd0cc487a5e24b27713bb
paper_authors.name: Kai Dang
paper_authors._id: 670dd0cc487a5e24b27713bc
paper_authors.name: An Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T06:43:16.208Z
paper_authors._id: 62088594a5943c8a8fc94560
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
paper_authors.fullname: An Yang
paper_authors.type: user
paper_authors.user: yangapku
paper_authors._id: 670dd0cc487a5e24b27713bd
paper_authors.name: Yuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T15:17:56.143Z
paper_authors._id: 670e57b3391f1a7021182bff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N0tuHZVz8KFPCv8G1qUX2.png
paper_authors.fullname: Yuan Wu
paper_authors.type: user
paper_authors.user: WhiteCatY
paper_authors._id: 670dd0cc487a5e24b27713be
paper_authors.name: Yuan Tian
paper_authors._id: 670dd0cc487a5e24b27713bf
paper_authors.name: Yi Chang
paper_authors._id: 670dd0cc487a5e24b27713c0
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:19:07.666Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: Rethinking Data Selection at Scale: Random Selection is Almost All You
    Need
paper_summary: Supervised fine-tuning (SFT) is crucial for aligning Large Language Models
  (LLMs) with human instructions. The primary goal during SFT is to select a
  small yet representative subset of training data from the larger pool, such
  that fine-tuning with this subset achieves results comparable to or even
  exceeding those obtained using the entire dataset. However, most existing data
  selection techniques are designed for small-scale data pools, which fail to
  meet the demands of real-world SFT scenarios. In this paper, we replicated
  several self-scoring methods those that do not rely on external model
  assistance on two million scale datasets, and found that nearly all methods
  struggled to significantly outperform random selection when dealing with such
  large-scale data pools. Moreover, our comparisons suggest that, during SFT,
  diversity in data selection is more critical than simply focusing on high
  quality data. We also analyzed the limitations of several current approaches,
  explaining why they perform poorly on large-scale datasets and why they are
  unsuitable for such contexts. Finally, we found that filtering data by token
  length offers a stable and efficient method for improving results. This
  approach, particularly when training on long text data, proves highly
  beneficial for relatively weaker base models, such as Llama3.
paper_ai_summary: Selecting representative subsets for supervised fine-tuning of LLMs from large datasets is challenging, and diversity in data selection is more critical than high data quality; token length filtering improves results.
paper_ai_keywords: Large Language Models Supervised fine-tuning self-scoring methods data selection large-scale datasets diversity base models token length filtering Llama3