title: Towards Open-ended Visual Quality Comparison
summary: Comparative settings (e.g. pairwise choice, listwise ranking) have been
  adopted by a wide range of subjective studies for image quality assessment
  (IQA), as it inherently standardizes the evaluation criteria across different
  observers and offer more clear-cut responses. In this work, we extend the edge
  of emerging large multi-modality models (LMMs) to further advance visual
  quality comparison into open-ended settings, that 1) can respond to open-range
  questions on quality comparison; 2) can provide detailed reasonings beyond
  direct answers. To this end, we propose the Co-Instruct. To train this
  first-of-its-kind open-source open-ended visual quality comparer, we collect
  the Co-Instruct-562K dataset, from two sources: (a) LMM-merged single image
  quality description, (b) GPT-4V "teacher" responses on unlabeled data.
  Furthermore, to better evaluate this setting, we propose the MICBench, the
  first benchmark on multi-image comparison for LMMs. We demonstrate that
  Co-Instruct not only achieves 30% higher superior accuracy than
  state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher),
  on both existing related benchmarks and the proposed MICBench. Our model is
  published at https://huggingface.co/q-future/co-instruct.
paper_authors._id: 65dd6b22538de889604760fa
paper_authors.name: Haoning Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T09:04:23.164Z
paper_authors._id: 63047ed2412a1b9d381b09c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
paper_authors.fullname: Haoning Wu, Teo
paper_authors.type: user
paper_authors.user: teowu
paper_authors._id: 65dd6b22538de889604760fb
paper_authors.name: Hanwei Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:47:00.109Z
paper_authors._id: 6437970dcd93f4c9a34e4c5a
paper_authors.avatarUrl: /avatars/5ef5cee3be250c600f0ed8cc5aad8dda.svg
paper_authors.fullname: Hanwei Zhu
paper_authors.type: user
paper_authors.user: HanweiZ
paper_authors._id: 65dd6b22538de889604760fc
paper_authors.name: Zicheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T09:04:24.805Z
paper_authors._id: 6526cc6bab4f5d98382f5603
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
paper_authors.fullname: Zicheng Zhang
paper_authors.type: user
paper_authors.user: zhangzicheng
paper_authors._id: 65dd6b22538de889604760fd
paper_authors.name: Erli Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:26:20.949Z
paper_authors._id: 639fd77cbeb95d698de6671e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639fd77cbeb95d698de6671e/nP7IkgzDa2NhyP5K2CTMi.jpeg
paper_authors.fullname: Zhang Erli
paper_authors.type: user
paper_authors.user: KarlZEL
paper_authors._id: 65dd6b22538de889604760fe
paper_authors.name: Chaofeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:27:35.212Z
paper_authors._id: 6304798d41387c7f117558f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6304798d41387c7f117558f7/_8OY8pVkApjg3q9Nnvjl2.jpeg
paper_authors.fullname: Chaofeng Chen
paper_authors.type: user
paper_authors.user: chaofengc
paper_authors._id: 65dd6b22538de889604760ff
paper_authors.name: Liang Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:27:42.113Z
paper_authors._id: 6582fe058fb810eaad787d72
paper_authors.avatarUrl: /avatars/54f03ee3e0c21664b20ea9b93ead4626.svg
paper_authors.fullname: Liang Liao
paper_authors.type: user
paper_authors.user: mercurialliao
paper_authors._id: 65dd6b22538de88960476100
paper_authors.name: Chunyi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:27:49.342Z
paper_authors._id: 644cf72f6d84e25fad73ae23
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644cf72f6d84e25fad73ae23/VF0V38obLNyKAbbTM65r8.jpeg
paper_authors.fullname: Chunyi Li
paper_authors.type: user
paper_authors.user: lcysyzxdxc
paper_authors._id: 65dd6b22538de88960476101
paper_authors.name: Annan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:27:56.973Z
paper_authors._id: 65534c8cd74d73c8c7ec6433
paper_authors.avatarUrl: /avatars/18c5fe8d1b5ba5a06f59e7f494dbd146.svg
paper_authors.fullname: Annan Wang
paper_authors.type: user
paper_authors.user: AnnanWang
paper_authors._id: 65dd6b22538de88960476102
paper_authors.name: Wenxiu Sun
paper_authors._id: 65dd6b22538de88960476103
paper_authors.name: Qiong Yan
paper_authors._id: 65dd6b22538de88960476104
paper_authors.name: Xiaohong Liu
paper_authors._id: 65dd6b22538de88960476105
paper_authors.name: Guangtao Zhai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:28:32.187Z
paper_authors._id: 65535b125413c1a54e6fb243
paper_authors.avatarUrl: /avatars/03bcf1d58865f5406aff49a415e78bdc.svg
paper_authors.fullname: Guangtao Zhai
paper_authors.type: user
paper_authors.user: GTZhai
paper_authors._id: 65dd6b22538de88960476106
paper_authors.name: Shiqi Wang
paper_authors._id: 65dd6b22538de88960476107
paper_authors.name: Weisi Lin
paper_title: Towards Open-ended Visual Quality Comparison
paper_summary: Comparative settings (e.g. pairwise choice, listwise ranking) have been
  adopted by a wide range of subjective studies for image quality assessment
  (IQA), as it inherently standardizes the evaluation criteria across different
  observers and offer more clear-cut responses. In this work, we extend the edge
  of emerging large multi-modality models (LMMs) to further advance visual
  quality comparison into open-ended settings, that 1) can respond to open-range
  questions on quality comparison; 2) can provide detailed reasonings beyond
  direct answers. To this end, we propose the Co-Instruct. To train this
  first-of-its-kind open-source open-ended visual quality comparer, we collect
  the Co-Instruct-562K dataset, from two sources: (a) LMM-merged single image
  quality description, (b) GPT-4V "teacher" responses on unlabeled data.
  Furthermore, to better evaluate this setting, we propose the MICBench, the
  first benchmark on multi-image comparison for LMMs. We demonstrate that
  Co-Instruct not only achieves 30% higher superior accuracy than
  state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher),
  on both existing related benchmarks and the proposed MICBench. Our model is
  published at https://huggingface.co/q-future/co-instruct.
paper_ai_keywords: large multi-modality models (LMMs) image quality assessment (IQA) pairwise choice listwise ranking open-ended settings open-range detailed reasonings Co-Instruct Co-Instruct-562K dataset GPT-4V single image quality description MICBench multi-image comparison benchmark