GenAI Arena: An Open Evaluation Platform for Generative Models
Generative AI has made remarkable strides to revolutionize fields such as
image and video generation. These advancements are driven by innovative
algorithms, architecture, and data. However, the rapid proliferation of
generative models has highlighted a critical gap: the absence of trustworthy
evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc
often fail to capture the nuanced quality and user satisfaction associated with
generative outputs. This paper proposes an open platform GenAI-Arena to
evaluate different image and video generative models, where users can actively
participate in evaluating these models. By leveraging collective user feedback
and votes, GenAI-Arena aims to provide a more democratic and accurate measure
of model performance. It covers three arenas for text-to-image generation,
text-to-video generation, and image editing respectively. Currently, we cover a
total of 27 open-source generative models. GenAI-Arena has been operating for
four months, amassing over 6000 votes from the community. We describe our
platform, analyze the data, and explain the statistical methods for ranking the
models. To further promote the research in building model-based evaluation
metrics, we release a cleaned version of our preference data for the three
tasks, namely GenAI-Bench. We prompt the existing multi-modal models like
Gemini, GPT-4o to mimic human voting. We compute the correlation between model
voting with human voting to understand their judging abilities. Our results
show existing multimodal models are still lagging in assessing the generated
visual content, even the best model GPT-4o only achieves a Pearson correlation
of 0.22 in the quality subscore, and behaves like random guessing in others.
66665b7a6fc71646eea75ce6
Dongfu Jiang
admin_assigned
2024-06-10T09:55:03.427Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
66665b7a6fc71646eea75ce7
Max Ku
claimed_verified
2024-06-10T10:03:16.479Z
631d760344503b7227837242
https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
Max Ku
user
vinesmsuic
66665b7a6fc71646eea75ce8
Tianle Li
admin_assigned
2024-06-10T09:56:00.478Z
63daf95a9f2687298a110386
/avatars/7e922ca76689de8e0e0c4af350115246.svg
Tianle LI
user
tianleliphoebe
66665b7a6fc71646eea75ce9
Yuansheng Ni
admin_assigned
2024-06-10T09:56:07.148Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
66665b7a6fc71646eea75cea
Shizhuo Sun
admin_assigned
2024-06-10T09:56:13.721Z
662d1410a314b134a2e64389
/avatars/70215fac94965f58750aa3d17defe124.svg
Shizhuo Sun
user
sunshizhuo
66665b7a6fc71646eea75ceb
Rongqi Fan
66665b7a6fc71646eea75cec
Wenhu Chen
admin_assigned
2024-06-10T09:56:36.776Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
GenAI Arena: An Open Evaluation Platform for Generative Models
Generative AI has made remarkable strides to revolutionize fields such as
image and video generation. These advancements are driven by innovative
algorithms, architecture, and data. However, the rapid proliferation of
generative models has highlighted a critical gap: the absence of trustworthy
evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc
often fail to capture the nuanced quality and user satisfaction associated with
generative outputs. This paper proposes an open platform GenAI-Arena to
evaluate different image and video generative models, where users can actively
participate in evaluating these models. By leveraging collective user feedback
and votes, GenAI-Arena aims to provide a more democratic and accurate measure
of model performance. It covers three arenas for text-to-image generation,
text-to-video generation, and image editing respectively. Currently, we cover a
total of 27 open-source generative models. GenAI-Arena has been operating for
four months, amassing over 6000 votes from the community. We describe our
platform, analyze the data, and explain the statistical methods for ranking the
models. To further promote the research in building model-based evaluation
metrics, we release a cleaned version of our preference data for the three
tasks, namely GenAI-Bench. We prompt the existing multi-modal models like
Gemini, GPT-4o to mimic human voting. We compute the correlation between model
voting with human voting to understand their judging abilities. Our results
show existing multimodal models are still lagging in assessing the generated
visual content, even the best model GPT-4o only achieves a Pearson correlation
of 0.22 in the quality subscore, and behaves like random guessing in others.
GenAI-Arena, an open platform, leverages user feedback to evaluate generative models across text-to-image, text-to-video, and image editing tasks, demonstrating existing multimodal models' shortcomings in assessing generated content quality.
generative AI image generation video generation evaluation metrics FID CLIP FVD user feedback text-to-image generation text-to-video generation image editing GenAI-Arena GenAI-Bench preference data multimodal models Pearson correlation