title: Evaluation Agent: Efficient and Promptable Evaluation Framework for
    Visual Generative Models
summary: Recent advancements in visual generative models have enabled high-quality
  image and video generation, opening diverse applications. However, evaluating
  these models often demands sampling hundreds or thousands of images or videos,
  making the process computationally expensive, especially for diffusion-based
  models with inherently slow sampling. Moreover, existing evaluation methods
  rely on rigid pipelines that overlook specific user needs and provide numerical
  results without clear explanations. In contrast, humans can quickly form
  impressions of a model's capabilities by observing only a few samples. To mimic
  this, we propose the Evaluation Agent framework, which employs human-like
  strategies for efficient, dynamic, multi-round evaluations using only a few
  samples per round, while offering detailed, user-tailored analyses. It offers
  four key advantages: 1) efficiency, 2) promptable evaluation tailored to
  diverse user needs, 3) explainability beyond single numerical scores, and 4)
  scalability across various models and tools. Experiments show that Evaluation
  Agent reduces evaluation time to 10% of traditional methods while delivering
  comparable results. The Evaluation Agent framework is fully open-sourced to
  advance research in visual generative models and their efficient evaluation.
paper_authors._id: 67610b5eb04baaf63514fcc9
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:03:36.826Z
paper_authors._id: 61f24cbb88b9b5abbe184a85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61f24cbb88b9b5abbe184a85/OvcJRU51yI8pdO77NBHLb.jpeg
paper_authors.fullname: zhangfan
paper_authors.type: user
paper_authors.user: Fan-s
paper_authors._id: 67610b5eb04baaf63514fcca
paper_authors.name: Shulin Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:23:16.414Z
paper_authors._id: 6658d01c6f1a71ba56d6c273
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tc4nZrMuZQLfgt5aVxtH4.jpeg
paper_authors.fullname: Tian Shulin
paper_authors.type: user
paper_authors.user: shulin16
paper_authors._id: 67610b5eb04baaf63514fccb
paper_authors.name: Ziqi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:23:04.978Z
paper_authors._id: 60efe7fa0d920bc7805cada5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
paper_authors.fullname: Ziqi Huang
paper_authors.type: user
paper_authors.user: Ziqi
paper_authors._id: 67610b5eb04baaf63514fccc
paper_authors.name: Yu Qiao
paper_authors._id: 67610b5eb04baaf63514fccd
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:22:56.546Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: Evaluation Agent: Efficient and Promptable Evaluation Framework for
    Visual Generative Models
paper_summary: Recent advancements in visual generative models have enabled high-quality
  image and video generation, opening diverse applications. However, evaluating
  these models often demands sampling hundreds or thousands of images or videos,
  making the process computationally expensive, especially for diffusion-based
  models with inherently slow sampling. Moreover, existing evaluation methods
  rely on rigid pipelines that overlook specific user needs and provide numerical
  results without clear explanations. In contrast, humans can quickly form
  impressions of a model's capabilities by observing only a few samples. To mimic
  this, we propose the Evaluation Agent framework, which employs human-like
  strategies for efficient, dynamic, multi-round evaluations using only a few
  samples per round, while offering detailed, user-tailored analyses. It offers
  four key advantages: 1) efficiency, 2) promptable evaluation tailored to
  diverse user needs, 3) explainability beyond single numerical scores, and 4)
  scalability across various models and tools. Experiments show that Evaluation
  Agent reduces evaluation time to 10% of traditional methods while delivering
  comparable results. The Evaluation Agent framework is fully open-sourced to
  advance research in visual generative models and their efficient evaluation.
paper_ai_summary: The Evaluation Agent framework efficiently evaluates visual generative models using human-like strategies, reducing time to 10% compared to traditional methods while maintaining quality.
paper_ai_keywords: diffusion-based models