title: K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models
    via K-wise Human Preferences
summary: The rapid advancement of visual generative models necessitates efficient and
  reliable evaluation methods. Arena platform, which gathers user votes on model
  comparisons, can rank models with human preferences. However, traditional Arena
  methods, while established, require an excessive number of comparisons for
  ranking to converge and are vulnerable to preference noise in voting,
  suggesting the need for better approaches tailored to contemporary evaluation
  challenges. In this paper, we introduce K-Sort Arena, an efficient and reliable
  platform based on a key insight: images and videos possess higher perceptual
  intuitiveness than texts, enabling rapid evaluation of multiple samples
  simultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowing
  K models to engage in free-for-all competitions, which yield much richer
  information than pairwise comparisons. To enhance the robustness of the system,
  we leverage probabilistic modeling and Bayesian updating techniques. We propose
  an exploration-exploitation-based matchmaking strategy to facilitate more
  informative comparisons. In our experiments, K-Sort Arena exhibits 16.3x faster
  convergence compared to the widely used ELO algorithm. To further validate the
  superiority and obtain a comprehensive leaderboard, we collect human feedback
  via crowdsourced evaluations of numerous cutting-edge text-to-image and
  text-to-video models. Thanks to its high efficiency, K-Sort Arena can
  continuously incorporate emerging models and update the leaderboard with
  minimal votes. Our project has undergone several months of internal testing and
  is now available at https://huggingface.co/spaces/ksort/K-Sort-Arena
paper_authors._id: 66cd35c000cddde87a5f720f
paper_authors.name: Zhikai Li
paper_authors._id: 66cd35c000cddde87a5f7210
paper_authors.name: Xuewen Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:14:04.667Z
paper_authors._id: 6480331be9ff472e30dc34d8
paper_authors.avatarUrl: /avatars/606ce64bfcfcb9afbaab9bf3287b9d5a.svg
paper_authors.fullname: liuxuewen
paper_authors.type: user
paper_authors.user: liiu
paper_authors._id: 66cd35c000cddde87a5f7211
paper_authors.name: Dongrong Fu
paper_authors._id: 66cd35c000cddde87a5f7212
paper_authors.name: Jianquan Li
paper_authors._id: 66cd35c000cddde87a5f7213
paper_authors.name: Qingyi Gu
paper_authors._id: 66cd35c000cddde87a5f7214
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:14:23.911Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 66cd35c000cddde87a5f7215
paper_authors.name: Zhen Dong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-09T19:36:01.297Z
paper_authors._id: 643ba2f725681c3afaa8f05e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
paper_authors.fullname: Zhen Dong
paper_authors.type: user
paper_authors.user: zhendongucb
paper_title: K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models
    via K-wise Human Preferences
paper_summary: The rapid advancement of visual generative models necessitates efficient and
  reliable evaluation methods. Arena platform, which gathers user votes on model
  comparisons, can rank models with human preferences. However, traditional Arena
  methods, while established, require an excessive number of comparisons for
  ranking to converge and are vulnerable to preference noise in voting,
  suggesting the need for better approaches tailored to contemporary evaluation
  challenges. In this paper, we introduce K-Sort Arena, an efficient and reliable
  platform based on a key insight: images and videos possess higher perceptual
  intuitiveness than texts, enabling rapid evaluation of multiple samples
  simultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowing
  K models to engage in free-for-all competitions, which yield much richer
  information than pairwise comparisons. To enhance the robustness of the system,
  we leverage probabilistic modeling and Bayesian updating techniques. We propose
  an exploration-exploitation-based matchmaking strategy to facilitate more
  informative comparisons. In our experiments, K-Sort Arena exhibits 16.3x faster
  convergence compared to the widely used ELO algorithm. To further validate the
  superiority and obtain a comprehensive leaderboard, we collect human feedback
  via crowdsourced evaluations of numerous cutting-edge text-to-image and
  text-to-video models. Thanks to its high efficiency, K-Sort Arena can
  continuously incorporate emerging models and update the leaderboard with
  minimal votes. Our project has undergone several months of internal testing and
  is now available at https://huggingface.co/spaces/ksort/K-Sort-Arena
paper_ai_summary: K-Sort Arena uses K-wise comparisons and probabilistic modeling to efficiently rank generative models based on human preference, showing significantly faster convergence than the ELO algorithm.
paper_ai_keywords: visual generative models Arena platform K-Sort Arena K-wise comparisons probabilistic modeling Bayesian updating exploration-exploitation-based matchmaking ELO algorithm text-to-image text-to-video