Evaluating Language Models as Synthetic Data Generators
Given the increasing use of synthetic data in language model (LM)
post-training, an LM's ability to generate high-quality data has become nearly
as crucial as its ability to solve problems directly. While prior works have
focused on developing effective data generation methods, they lack systematic
comparison of different LMs as data generators in a unified setting. To address
this gap, we propose AgoraBench, a benchmark that provides standardized
settings and metrics to evaluate LMs' data generation abilities. Through
synthesizing 1.26 million training instances using 6 LMs and training 99
student models, we uncover key insights about LMs' data generation
capabilities. First, we observe that LMs exhibit distinct strengths. For
instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet
performs better at enhancing existing ones. Furthermore, our analysis reveals
that an LM's data generation ability doesn't necessarily correlate with its
problem-solving ability. Instead, multiple intrinsic features of data
quality-including response quality, perplexity, and instruction
difficulty-collectively serve as better indicators. Finally, we demonstrate
that strategic choices in output format and cost-conscious model selection
significantly impact data generation effectiveness.
67525bedeeb66c5ab7dc99fa
Seungone Kim
claimed_verified
2024-12-06T09:09:33.388Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
67525bedeeb66c5ab7dc99fb
Juyoung Suk
admin_assigned
2024-12-06T17:23:46.560Z
6138cc1306dd10833d2db64b
https://cdn-avatars.huggingface.co/v1/production/uploads/6138cc1306dd10833d2db64b/IRX4y-8M4YlzR_8jOwkKp.jpeg
Juyoung Suk
user
juyoungml
67525bedeeb66c5ab7dc99fc
Xiang Yue
admin_assigned
2024-12-06T17:23:39.076Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
67525bedeeb66c5ab7dc99fd
Vijay Viswanathan
admin_assigned
2024-12-06T17:23:31.437Z
603c44e677a2a899efe25828
https://cdn-avatars.huggingface.co/v1/production/uploads/1614562504843-noauth.jpeg
Vijay Viswanathan
user
viswavi
67525bedeeb66c5ab7dc99fe
Seongyun Lee
admin_assigned
2024-12-06T17:23:18.129Z
6550c4f27bbfce1878f5f280
/avatars/0ecedbcd8a55b2c4abd1da9e741a6652.svg
seongyun_lee
user
Seongyun
67525bedeeb66c5ab7dc99ff
Yizhong Wang
admin_assigned
2024-12-06T17:22:46.849Z
6269be67d1ac0cde592aba29
https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
Yizhong Wang
user
yizhongw
67525bedeeb66c5ab7dc9a00
Kiril Gashteovski
admin_assigned
2024-12-06T17:22:41.109Z
638763415c68cf2713b8ad7c
/avatars/3113c3e71caa5cd5b6f8ce9c28241bc3.svg
Kiril Gashteovski
user
kgashteo
67525bedeeb66c5ab7dc9a01
Carolin Lawrence
67525bedeeb66c5ab7dc9a02
Sean Welleck
admin_assigned
2024-12-06T17:22:32.024Z
63e3f3c59db5da2dc1ef6889
/avatars/f7546f57a5fd69bc99ff1640cc4a4853.svg
Sean Welleck
user
wellecks
67525bedeeb66c5ab7dc9a03
Graham Neubig
admin_assigned
2024-12-06T17:22:07.993Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
Evaluating Language Models as Synthetic Data Generators
Given the increasing use of synthetic data in language model (LM)
post-training, an LM's ability to generate high-quality data has become nearly
as crucial as its ability to solve problems directly. While prior works have
focused on developing effective data generation methods, they lack systematic
comparison of different LMs as data generators in a unified setting. To address
this gap, we propose AgoraBench, a benchmark that provides standardized
settings and metrics to evaluate LMs' data generation abilities. Through
synthesizing 1.26 million training instances using 6 LMs and training 99
student models, we uncover key insights about LMs' data generation
capabilities. First, we observe that LMs exhibit distinct strengths. For
instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet
performs better at enhancing existing ones. Furthermore, our analysis reveals
that an LM's data generation ability doesn't necessarily correlate with its
problem-solving ability. Instead, multiple intrinsic features of data
quality-including response quality, perplexity, and instruction
difficulty-collectively serve as better indicators. Finally, we demonstrate
that strategic choices in output format and cost-conscious model selection
significantly impact data generation effectiveness.
AgoraBench evaluates language models' data generation abilities, revealing their distinct strengths and demonstrating various factors that impact effectiveness.
language models data generation AgoraBench standardized settings metrics training instances student models GPT-4o Claude-3.5-Sonnet response quality perplexity instruction difficulty output format model selection