title: Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and
    Accountability
summary: High-quality datasets are fundamental to training and evaluating machine
  learning models, yet their creation-especially with accurate human
  annotations-remains a significant challenge. Many dataset paper submissions
  lack originality, diversity, or rigorous quality control, and these
  shortcomings are often overlooked during peer review. Submissions also
  frequently omit essential details about dataset construction and properties.
  While existing tools such as datasheets aim to promote transparency, they are
  largely descriptive and do not provide standardized, measurable methods for
  evaluating data quality. Similarly, metadata requirements at conferences
  promote accountability but are inconsistently enforced. To address these
  limitations, this position paper advocates for the integration of systematic,
  rubric-based evaluation metrics into the dataset review process-particularly as
  submission volumes continue to grow. We also explore scalable, cost-effective
  methods for synthetic data generation, including dedicated tools and
  LLM-as-a-judge approaches, to support more efficient evaluation. As a call to
  action, we introduce DataRubrics, a structured framework for assessing the
  quality of both human- and model-generated datasets. Leveraging recent advances
  in LLM-based evaluation, DataRubrics offers a reproducible, scalable, and
  actionable solution for dataset quality assessment, enabling both authors and
  reviewers to uphold higher standards in data-centric research. We also release
  code to support reproducibility of LLM-based evaluations at
  https://github.com/datarubrics/datarubrics.
paper_authors._id: 683fb99c7c8d720be438000a
paper_authors.name: Genta Indra Winata
paper_authors._id: 683fb99c7c8d720be438000b
paper_authors.name: David Anugraha
paper_authors._id: 683fb99c7c8d720be438000c
paper_authors.name: Emmy Liu
paper_authors._id: 683fb99c7c8d720be438000d
paper_authors.name: Alham Fikri Aji
paper_authors._id: 683fb99c7c8d720be438000e
paper_authors.name: Shou-Yi Hung
paper_authors._id: 683fb99c7c8d720be438000f
paper_authors.name: Aditya Parashar
paper_authors._id: 683fb99c7c8d720be4380010
paper_authors.name: Patrick Amadeus Irawan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:55:30.937Z
paper_authors._id: 64d1e3a87e20ec9ea0020d03
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d1e3a87e20ec9ea0020d03/xm-afh1AaqS0e9qpaPo7y.jpeg
paper_authors.fullname: Patrick Amadeus Irawan
paper_authors.type: user
paper_authors.user: patrickamadeus
paper_authors._id: 683fb99c7c8d720be4380011
paper_authors.name: Ruochen Zhang
paper_authors._id: 683fb99c7c8d720be4380012
paper_authors.name: Zheng-Xin Yong
paper_authors._id: 683fb99c7c8d720be4380013
paper_authors.name: Jan Christian Blaise Cruz
paper_authors._id: 683fb99c7c8d720be4380014
paper_authors.name: Niklas Muennighoff
paper_authors._id: 683fb99c7c8d720be4380015
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:55:22.158Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 683fb99c7c8d720be4380016
paper_authors.name: Hanyang Zhao
paper_authors._id: 683fb99c7c8d720be4380017
paper_authors.name: Sudipta Kar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:55:28.286Z
paper_authors._id: 63139ff6b46fc4e24332fa84
paper_authors.avatarUrl: /avatars/ee6923a7cb218f22535064a87761e497.svg
paper_authors.fullname: Sudipta Kar
paper_authors.type: user
paper_authors.user: cryptexcode
paper_authors._id: 683fb99c7c8d720be4380018
paper_authors.name: Kezia Erina Suryoraharjo
paper_authors._id: 683fb99c7c8d720be4380019
paper_authors.name: M. Farid Adilazuarda
paper_authors._id: 683fb99c7c8d720be438001a
paper_authors.name: En-Shiun Annie Lee
paper_authors._id: 683fb99c7c8d720be438001b
paper_authors.name: Ayu Purwarianti
paper_authors._id: 683fb99c7c8d720be438001c
paper_authors.name: Derry Tanti Wijaya
paper_authors._id: 683fb99c7c8d720be438001d
paper_authors.name: Monojit Choudhury
paper_title: Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and
    Accountability
paper_summary: High-quality datasets are fundamental to training and evaluating machine
  learning models, yet their creation-especially with accurate human
  annotations-remains a significant challenge. Many dataset paper submissions
  lack originality, diversity, or rigorous quality control, and these
  shortcomings are often overlooked during peer review. Submissions also
  frequently omit essential details about dataset construction and properties.
  While existing tools such as datasheets aim to promote transparency, they are
  largely descriptive and do not provide standardized, measurable methods for
  evaluating data quality. Similarly, metadata requirements at conferences
  promote accountability but are inconsistently enforced. To address these
  limitations, this position paper advocates for the integration of systematic,
  rubric-based evaluation metrics into the dataset review process-particularly as
  submission volumes continue to grow. We also explore scalable, cost-effective
  methods for synthetic data generation, including dedicated tools and
  LLM-as-a-judge approaches, to support more efficient evaluation. As a call to
  action, we introduce DataRubrics, a structured framework for assessing the
  quality of both human- and model-generated datasets. Leveraging recent advances
  in LLM-based evaluation, DataRubrics offers a reproducible, scalable, and
  actionable solution for dataset quality assessment, enabling both authors and
  reviewers to uphold higher standards in data-centric research. We also release
  code to support reproducibility of LLM-based evaluations at
  https://github.com/datarubrics/datarubrics.