title: Evaluating Language Models as Synthetic Data Generators
summary: Given the increasing use of synthetic data in language model (LM)
  post-training, an LM's ability to generate high-quality data has become nearly
  as crucial as its ability to solve problems directly. While prior works have
  focused on developing effective data generation methods, they lack systematic
  comparison of different LMs as data generators in a unified setting. To address
  this gap, we propose AgoraBench, a benchmark that provides standardized
  settings and metrics to evaluate LMs' data generation abilities. Through
  synthesizing 1.26 million training instances using 6 LMs and training 99
  student models, we uncover key insights about LMs' data generation
  capabilities. First, we observe that LMs exhibit distinct strengths. For
  instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet
  performs better at enhancing existing ones. Furthermore, our analysis reveals
  that an LM's data generation ability doesn't necessarily correlate with its
  problem-solving ability. Instead, multiple intrinsic features of data
  quality-including response quality, perplexity, and instruction
  difficulty-collectively serve as better indicators. Finally, we demonstrate
  that strategic choices in output format and cost-conscious model selection
  significantly impact data generation effectiveness.
paper_authors._id: 67525bedeeb66c5ab7dc99fa
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:33.388Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 67525bedeeb66c5ab7dc99fb
paper_authors.name: Juyoung Suk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:23:46.560Z
paper_authors._id: 6138cc1306dd10833d2db64b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6138cc1306dd10833d2db64b/IRX4y-8M4YlzR_8jOwkKp.jpeg
paper_authors.fullname: Juyoung Suk
paper_authors.type: user
paper_authors.user: juyoungml
paper_authors._id: 67525bedeeb66c5ab7dc99fc
paper_authors.name: Xiang Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:23:39.076Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 67525bedeeb66c5ab7dc99fd
paper_authors.name: Vijay Viswanathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:23:31.437Z
paper_authors._id: 603c44e677a2a899efe25828
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1614562504843-noauth.jpeg
paper_authors.fullname: Vijay Viswanathan
paper_authors.type: user
paper_authors.user: viswavi
paper_authors._id: 67525bedeeb66c5ab7dc99fe
paper_authors.name: Seongyun Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:23:18.129Z
paper_authors._id: 6550c4f27bbfce1878f5f280
paper_authors.avatarUrl: /avatars/0ecedbcd8a55b2c4abd1da9e741a6652.svg
paper_authors.fullname: seongyun_lee
paper_authors.type: user
paper_authors.user: Seongyun
paper_authors._id: 67525bedeeb66c5ab7dc99ff
paper_authors.name: Yizhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:22:46.849Z
paper_authors._id: 6269be67d1ac0cde592aba29
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
paper_authors.fullname: Yizhong Wang
paper_authors.type: user
paper_authors.user: yizhongw
paper_authors._id: 67525bedeeb66c5ab7dc9a00
paper_authors.name: Kiril Gashteovski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:22:41.109Z
paper_authors._id: 638763415c68cf2713b8ad7c
paper_authors.avatarUrl: /avatars/3113c3e71caa5cd5b6f8ce9c28241bc3.svg
paper_authors.fullname: Kiril Gashteovski
paper_authors.type: user
paper_authors.user: kgashteo
paper_authors._id: 67525bedeeb66c5ab7dc9a01
paper_authors.name: Carolin Lawrence
paper_authors._id: 67525bedeeb66c5ab7dc9a02
paper_authors.name: Sean Welleck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:22:32.024Z
paper_authors._id: 63e3f3c59db5da2dc1ef6889
paper_authors.avatarUrl: /avatars/f7546f57a5fd69bc99ff1640cc4a4853.svg
paper_authors.fullname: Sean Welleck
paper_authors.type: user
paper_authors.user: wellecks
paper_authors._id: 67525bedeeb66c5ab7dc9a03
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:22:07.993Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_title: Evaluating Language Models as Synthetic Data Generators
paper_summary: Given the increasing use of synthetic data in language model (LM)
  post-training, an LM's ability to generate high-quality data has become nearly
  as crucial as its ability to solve problems directly. While prior works have
  focused on developing effective data generation methods, they lack systematic
  comparison of different LMs as data generators in a unified setting. To address
  this gap, we propose AgoraBench, a benchmark that provides standardized
  settings and metrics to evaluate LMs' data generation abilities. Through
  synthesizing 1.26 million training instances using 6 LMs and training 99
  student models, we uncover key insights about LMs' data generation
  capabilities. First, we observe that LMs exhibit distinct strengths. For
  instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet
  performs better at enhancing existing ones. Furthermore, our analysis reveals
  that an LM's data generation ability doesn't necessarily correlate with its
  problem-solving ability. Instead, multiple intrinsic features of data
  quality-including response quality, perplexity, and instruction
  difficulty-collectively serve as better indicators. Finally, we demonstrate
  that strategic choices in output format and cost-conscious model selection
  significantly impact data generation effectiveness.
paper_ai_summary: AgoraBench evaluates language models' data generation abilities, revealing their distinct strengths and demonstrating various factors that impact effectiveness.
paper_ai_keywords: language models data generation AgoraBench standardized settings metrics training instances student models GPT-4o Claude-3.5-Sonnet response quality perplexity instruction difficulty output format model selection