title: GenAI Arena: An Open Evaluation Platform for Generative Models
summary: Generative AI has made remarkable strides to revolutionize fields such as
  image and video generation. These advancements are driven by innovative
  algorithms, architecture, and data. However, the rapid proliferation of
  generative models has highlighted a critical gap: the absence of trustworthy
  evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc
  often fail to capture the nuanced quality and user satisfaction associated with
  generative outputs. This paper proposes an open platform GenAI-Arena to
  evaluate different image and video generative models, where users can actively
  participate in evaluating these models. By leveraging collective user feedback
  and votes, GenAI-Arena aims to provide a more democratic and accurate measure
  of model performance. It covers three arenas for text-to-image generation,
  text-to-video generation, and image editing respectively. Currently, we cover a
  total of 27 open-source generative models. GenAI-Arena has been operating for
  four months, amassing over 6000 votes from the community. We describe our
  platform, analyze the data, and explain the statistical methods for ranking the
  models. To further promote the research in building model-based evaluation
  metrics, we release a cleaned version of our preference data for the three
  tasks, namely GenAI-Bench. We prompt the existing multi-modal models like
  Gemini, GPT-4o to mimic human voting. We compute the correlation between model
  voting with human voting to understand their judging abilities. Our results
  show existing multimodal models are still lagging in assessing the generated
  visual content, even the best model GPT-4o only achieves a Pearson correlation
  of 0.22 in the quality subscore, and behaves like random guessing in others.
paper_authors._id: 66665b7a6fc71646eea75ce6
paper_authors.name: Dongfu Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T09:55:03.427Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 66665b7a6fc71646eea75ce7
paper_authors.name: Max Ku
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-10T10:03:16.479Z
paper_authors._id: 631d760344503b7227837242
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
paper_authors.fullname: Max Ku
paper_authors.type: user
paper_authors.user: vinesmsuic
paper_authors._id: 66665b7a6fc71646eea75ce8
paper_authors.name: Tianle Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T09:56:00.478Z
paper_authors._id: 63daf95a9f2687298a110386
paper_authors.avatarUrl: /avatars/7e922ca76689de8e0e0c4af350115246.svg
paper_authors.fullname: Tianle LI
paper_authors.type: user
paper_authors.user: tianleliphoebe
paper_authors._id: 66665b7a6fc71646eea75ce9
paper_authors.name: Yuansheng Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T09:56:07.148Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 66665b7a6fc71646eea75cea
paper_authors.name: Shizhuo Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T09:56:13.721Z
paper_authors._id: 662d1410a314b134a2e64389
paper_authors.avatarUrl: /avatars/70215fac94965f58750aa3d17defe124.svg
paper_authors.fullname: Shizhuo Sun
paper_authors.type: user
paper_authors.user: sunshizhuo
paper_authors._id: 66665b7a6fc71646eea75ceb
paper_authors.name: Rongqi Fan
paper_authors._id: 66665b7a6fc71646eea75cec
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T09:56:36.776Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: GenAI Arena: An Open Evaluation Platform for Generative Models
paper_summary: Generative AI has made remarkable strides to revolutionize fields such as
  image and video generation. These advancements are driven by innovative
  algorithms, architecture, and data. However, the rapid proliferation of
  generative models has highlighted a critical gap: the absence of trustworthy
  evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc
  often fail to capture the nuanced quality and user satisfaction associated with
  generative outputs. This paper proposes an open platform GenAI-Arena to
  evaluate different image and video generative models, where users can actively
  participate in evaluating these models. By leveraging collective user feedback
  and votes, GenAI-Arena aims to provide a more democratic and accurate measure
  of model performance. It covers three arenas for text-to-image generation,
  text-to-video generation, and image editing respectively. Currently, we cover a
  total of 27 open-source generative models. GenAI-Arena has been operating for
  four months, amassing over 6000 votes from the community. We describe our
  platform, analyze the data, and explain the statistical methods for ranking the
  models. To further promote the research in building model-based evaluation
  metrics, we release a cleaned version of our preference data for the three
  tasks, namely GenAI-Bench. We prompt the existing multi-modal models like
  Gemini, GPT-4o to mimic human voting. We compute the correlation between model
  voting with human voting to understand their judging abilities. Our results
  show existing multimodal models are still lagging in assessing the generated
  visual content, even the best model GPT-4o only achieves a Pearson correlation
  of 0.22 in the quality subscore, and behaves like random guessing in others.
paper_ai_keywords: FID CLIP FVD GenAI-Arena text-to-image generation text-to-video generation image editing GenAI-Bench multimodal models Gemini GPT-4o Pearson correlation