K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models
  via K-wise Human Preferences
The rapid advancement of visual generative models necessitates efficient and
reliable evaluation methods. Arena platform, which gathers user votes on model
comparisons, can rank models with human preferences. However, traditional Arena
methods, while established, require an excessive number of comparisons for
ranking to converge and are vulnerable to preference noise in voting,
suggesting the need for better approaches tailored to contemporary evaluation
challenges. In this paper, we introduce K-Sort Arena, an efficient and reliable
platform based on a key insight: images and videos possess higher perceptual
intuitiveness than texts, enabling rapid evaluation of multiple samples
simultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowing
K models to engage in free-for-all competitions, which yield much richer
information than pairwise comparisons. To enhance the robustness of the system,
we leverage probabilistic modeling and Bayesian updating techniques. We propose
an exploration-exploitation-based matchmaking strategy to facilitate more
informative comparisons. In our experiments, K-Sort Arena exhibits 16.3x faster
convergence compared to the widely used ELO algorithm. To further validate the
superiority and obtain a comprehensive leaderboard, we collect human feedback
via crowdsourced evaluations of numerous cutting-edge text-to-image and
text-to-video models. Thanks to its high efficiency, K-Sort Arena can
continuously incorporate emerging models and update the leaderboard with
minimal votes. Our project has undergone several months of internal testing and
is now available at https://huggingface.co/spaces/ksort/K-Sort-Arena
66cd35c000cddde87a5f720f
Zhikai Li
66cd35c000cddde87a5f7210
Xuewen Liu
admin_assigned
2024-08-27T12:14:04.667Z
6480331be9ff472e30dc34d8
/avatars/606ce64bfcfcb9afbaab9bf3287b9d5a.svg
liuxuewen
user
liiu
66cd35c000cddde87a5f7211
Dongrong Fu
66cd35c000cddde87a5f7212
Jianquan Li
66cd35c000cddde87a5f7213
Qingyi Gu
66cd35c000cddde87a5f7214
Kurt Keutzer
admin_assigned
2024-08-27T12:14:23.911Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
66cd35c000cddde87a5f7215
Zhen Dong
extracted_confirmed
2024-10-09T19:36:01.297Z
643ba2f725681c3afaa8f05e
https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
Zhen Dong
user
zhendongucb
K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models
  via K-wise Human Preferences
The rapid advancement of visual generative models necessitates efficient and
reliable evaluation methods. Arena platform, which gathers user votes on model
comparisons, can rank models with human preferences. However, traditional Arena
methods, while established, require an excessive number of comparisons for
ranking to converge and are vulnerable to preference noise in voting,
suggesting the need for better approaches tailored to contemporary evaluation
challenges. In this paper, we introduce K-Sort Arena, an efficient and reliable
platform based on a key insight: images and videos possess higher perceptual
intuitiveness than texts, enabling rapid evaluation of multiple samples
simultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowing
K models to engage in free-for-all competitions, which yield much richer
information than pairwise comparisons. To enhance the robustness of the system,
we leverage probabilistic modeling and Bayesian updating techniques. We propose
an exploration-exploitation-based matchmaking strategy to facilitate more
informative comparisons. In our experiments, K-Sort Arena exhibits 16.3x faster
convergence compared to the widely used ELO algorithm. To further validate the
superiority and obtain a comprehensive leaderboard, we collect human feedback
via crowdsourced evaluations of numerous cutting-edge text-to-image and
text-to-video models. Thanks to its high efficiency, K-Sort Arena can
continuously incorporate emerging models and update the leaderboard with
minimal votes. Our project has undergone several months of internal testing and
is now available at https://huggingface.co/spaces/ksort/K-Sort-Arena
K-Sort Arena uses K-wise comparisons and probabilistic modeling to efficiently rank generative models based on human preference, showing significantly faster convergence than the ELO algorithm.
visual generative models Arena platform K-Sort Arena K-wise comparisons probabilistic modeling Bayesian updating exploration-exploitation-based matchmaking ELO algorithm text-to-image text-to-video