title: Evaluating Text Creativity across Diverse Domains: A Dataset and Large
    Language Model Evaluator
summary: Creativity evaluation remains a challenging frontier for large language
  models (LLMs). Current evaluations heavily rely on inefficient and costly human
  judgments, hindering progress in enhancing machine creativity. While automated
  methods exist, ranging from psychological testing to heuristic- or
  prompting-based approaches, they often lack generalizability or alignment with
  human judgment. To address these issues, in this paper, we propose a novel
  pairwise-comparison framework for assessing textual creativity, leveraging
  shared contextual instructions to improve evaluation consistency. We introduce
  CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic
  creative instruction-response pairs spanning diverse open-domain tasks. Through
  training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval
  demonstrates remarkable superiority over existing methods in alignment with
  human judgments. Experimental results underscore the indispensable significance
  of integrating both human-generated and synthetic data in training highly
  robust evaluators, and showcase the practical utility of CrEval in boosting the
  creativity of LLMs. We will release all data, code, and models publicly soon to
  support further research.
paper_authors._id: 68392fc4b6280677f75e6194
paper_authors.name: Qian Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:53:24.101Z
paper_authors._id: 5fbdf878485ef14d9a960f4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634724388998-5fbdf878485ef14d9a960f4d.jpeg
paper_authors.fullname: Qian Cao
paper_authors.type: user
paper_authors.user: Aman
paper_authors._id: 68392fc4b6280677f75e6195
paper_authors.name: Xiting Wang
paper_authors._id: 68392fc4b6280677f75e6196
paper_authors.name: Yuzhuo Yuan
paper_authors._id: 68392fc4b6280677f75e6197
paper_authors.name: Yahui Liu
paper_authors._id: 68392fc4b6280677f75e6198
paper_authors.name: Fang Luo
paper_authors._id: 68392fc4b6280677f75e6199
paper_authors.name: Ruihua Song
paper_title: Evaluating Text Creativity across Diverse Domains: A Dataset and Large
    Language Model Evaluator
paper_summary: Creativity evaluation remains a challenging frontier for large language
  models (LLMs). Current evaluations heavily rely on inefficient and costly human
  judgments, hindering progress in enhancing machine creativity. While automated
  methods exist, ranging from psychological testing to heuristic- or
  prompting-based approaches, they often lack generalizability or alignment with
  human judgment. To address these issues, in this paper, we propose a novel
  pairwise-comparison framework for assessing textual creativity, leveraging
  shared contextual instructions to improve evaluation consistency. We introduce
  CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic
  creative instruction-response pairs spanning diverse open-domain tasks. Through
  training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval
  demonstrates remarkable superiority over existing methods in alignment with
  human judgments. Experimental results underscore the indispensable significance
  of integrating both human-generated and synthetic data in training highly
  robust evaluators, and showcase the practical utility of CrEval in boosting the
  creativity of LLMs. We will release all data, code, and models publicly soon to
  support further research.
paper_ai_summary: A novel pairwise-comparison framework using CreataSet dataset trains CrEval, an LLM-based evaluator that significantly improves the assessment of textual creativity aligned with human judgments.
paper_ai_keywords: large language models LLMs automated methods heuristic-based approaches prompting-based approaches pairwise-comparison framework shared contextual instructions CreataSet synthetic creative instruction-response pairs open-domain tasks human-level instructions CrEval human judgments robust evaluators creativity enhancement