title: Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining
summary: Efficient data selection is crucial to accelerate the pretraining of large
  language models (LLMs). While various methods have been proposed to enhance
  data efficiency, limited research has addressed the inherent conflicts between
  these approaches to achieve optimal data selection for LLM pretraining. To
  tackle this problem, we propose a novel multi-agent collaborative data
  selection mechanism. In this framework, each data selection method serves as an
  independent agent, and an agent console is designed to dynamically integrate
  the information from all agents throughout the LLM training process. We conduct
  extensive empirical studies to evaluate our multi-agent framework. The
  experimental results demonstrate that our approach significantly improves data
  efficiency, accelerates convergence in LLM training, and achieves an average
  performance gain of 10.5% across multiple language model benchmarks compared to
  the state-of-the-art methods.
paper_authors._id: 670cab96200342b4d2c77ff5
paper_authors.name: Tianyi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:47:44.353Z
paper_authors._id: 641802c31f1f3b0fa812225c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641802c31f1f3b0fa812225c/i-OFWm8d37ofsmuvvAKYR.png
paper_authors.fullname: becca_bai
paper_authors.type: user
paper_authors.user: beccabai
paper_authors._id: 670cab96200342b4d2c77ff6
paper_authors.name: Ling Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:37:15.149Z
paper_authors._id: 64fde4e252e82dd432b74ce9
paper_authors.avatarUrl: /avatars/061a69d858b86d1600be916122cae7fc.svg
paper_authors.fullname: Ling Yang
paper_authors.type: user
paper_authors.user: Lingaaaaaaa
paper_authors._id: 670cab96200342b4d2c77ff7
paper_authors.name: Zhen Hao Wong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:38:20.132Z
paper_authors._id: 65d3149f0545ab7c11568b2b
paper_authors.avatarUrl: /avatars/02f36d8320b8331849e7d7f40e509ca8.svg
paper_authors.fullname: Zhen Hao Wong
paper_authors.type: user
paper_authors.user: aaron1141
paper_authors._id: 670cab96200342b4d2c77ff8
paper_authors.name: Jiahui Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:38:52.705Z
paper_authors._id: 64aa3d4b329f5e6e780dce4a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5F_eFJ8ZZOgDxrOHSFyzJ.png
paper_authors.fullname: Jiahui Peng
paper_authors.type: user
paper_authors.user: Jiahui2023
paper_authors._id: 670cab96200342b4d2c77ff9
paper_authors.name: Xinlin Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:38:46.006Z
paper_authors._id: 63567f00f44a9f0f5f885221
paper_authors.avatarUrl: /avatars/bc41b72399b7a7fe9293cde62f0577a0.svg
paper_authors.fullname: Xinlin Zhuang
paper_authors.type: user
paper_authors.user: Mihara-bot
paper_authors._id: 670cab96200342b4d2c77ffa
paper_authors.name: Chi Zhang
paper_authors._id: 670cab96200342b4d2c77ffb
paper_authors.name: Lijun Wu
paper_authors._id: 670cab96200342b4d2c77ffc
paper_authors.name: Qiu Jiantao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:39:34.816Z
paper_authors._id: 641bbd19af42e9b7dd4638ce
paper_authors.avatarUrl: /avatars/8e7819fa76650cab30e60063e8d3a217.svg
paper_authors.fullname: Qiu Jiantao
paper_authors.type: user
paper_authors.user: qiujiantao
paper_authors._id: 670cab96200342b4d2c77ffd
paper_authors.name: Wentao Zhang
paper_authors._id: 670cab96200342b4d2c77ffe
paper_authors.name: Binhang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:40:04.373Z
paper_authors._id: 6307c8dda642348dc5e71c20
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
paper_authors.fullname: Binhang Yuan
paper_authors.type: user
paper_authors.user: biyuan
paper_authors._id: 670cab96200342b4d2c77fff
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:39:58.611Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_title: Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining
paper_summary: Efficient data selection is crucial to accelerate the pretraining of large
  language models (LLMs). While various methods have been proposed to enhance
  data efficiency, limited research has addressed the inherent conflicts between
  these approaches to achieve optimal data selection for LLM pretraining. To
  tackle this problem, we propose a novel multi-agent collaborative data
  selection mechanism. In this framework, each data selection method serves as an
  independent agent, and an agent console is designed to dynamically integrate
  the information from all agents throughout the LLM training process. We conduct
  extensive empirical studies to evaluate our multi-agent framework. The
  experimental results demonstrate that our approach significantly improves data
  efficiency, accelerates convergence in LLM training, and achieves an average
  performance gain of 10.5% across multiple language model benchmarks compared to
  the state-of-the-art methods.
paper_ai_summary: A multi-agent collaborative mechanism enhances data selection for pretraining large language models, improving efficiency and performance.
paper_ai_keywords: multi-agent collaborative data selection large language models LLM pretraining data efficiency convergence performance gain