title: Q-Instruct: Improving Low-level Visual Abilities for Multi-modality
    Foundation Models
summary: Multi-modality foundation models, as represented by GPT-4V, have brought a
  new paradigm for low-level visual perception and understanding tasks, that can
  respond to a broad range of natural human instructions in a model. While
  existing foundation models have shown exciting potentials on low-level visual
  tasks, their related abilities are still preliminary and need to be improved.
  In order to enhance these models, we conduct a large-scale subjective
  experiment collecting a vast number of real human feedbacks on low-level
  vision. Each feedback follows a pathway that starts with a detailed description
  on the low-level visual appearance (*e.g. clarity, color, brightness* of an
  image, and ends with an overall conclusion, with an average length of 45 words.
  The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
  18,973 images with diverse low-level appearance. Moreover, to enable foundation
  models to robustly respond to diverse types of questions, we design a
  GPT-participated conversion to process these feedbacks into diverse-format 200K
  instruction-response pairs. Experimental results indicate that the
  **Q-Instruct** consistently elevates low-level perception and understanding
  abilities across several foundational models. We anticipate that our datasets
  can pave the way for a future that general intelligence can perceive,
  understand low-level visual appearance and evaluate visual quality like a
  human. Our dataset, model zoo, and demo is published at:
  https://q-future.github.io/Q-Instruct.
paper_authors._id: 6552e2c9812ed6f6ea59459b
paper_authors.name: Haoning Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T09:36:57.376Z
paper_authors._id: 63047ed2412a1b9d381b09c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
paper_authors.fullname: Haoning Wu, Teo
paper_authors.type: user
paper_authors.user: teowu
paper_authors._id: 6552e2c9812ed6f6ea59459c
paper_authors.name: Zicheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T10:07:11.940Z
paper_authors._id: 6526cc6bab4f5d98382f5603
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
paper_authors.fullname: Zicheng Zhang
paper_authors.type: user
paper_authors.user: zhangzicheng
paper_authors._id: 6552e2c9812ed6f6ea59459d
paper_authors.name: Erli Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T09:50:40.732Z
paper_authors._id: 639fd77cbeb95d698de6671e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639fd77cbeb95d698de6671e/nP7IkgzDa2NhyP5K2CTMi.jpeg
paper_authors.fullname: Zhang Erli
paper_authors.type: user
paper_authors.user: KarlZEL
paper_authors._id: 6552e2c9812ed6f6ea59459e
paper_authors.name: Chaofeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T09:50:52.872Z
paper_authors._id: 64bb4468b567ae97c3023293
paper_authors.avatarUrl: /avatars/e575b7e0198b9622d4be3cb04c677d92.svg
paper_authors.fullname: chenchaofeng
paper_authors.type: user
paper_authors.user: chenchaofeng
paper_authors._id: 6552e2c9812ed6f6ea59459f
paper_authors.name: Liang Liao
paper_authors._id: 6552e2c9812ed6f6ea5945a0
paper_authors.name: Annan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T10:34:07.136Z
paper_authors._id: 65534c8cd74d73c8c7ec6433
paper_authors.avatarUrl: /avatars/18c5fe8d1b5ba5a06f59e7f494dbd146.svg
paper_authors.fullname: Annan Wang
paper_authors.type: user
paper_authors.user: AnnanWang
paper_authors._id: 6552e2c9812ed6f6ea5945a1
paper_authors.name: Kaixin Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T10:34:10.203Z
paper_authors._id: 65534cf5b0b46eaf2f9499b3
paper_authors.avatarUrl: /avatars/4e681322d2e8c1c705589596bf0c3650.svg
paper_authors.fullname: Kaixin Xu
paper_authors.type: user
paper_authors.user: kartmannXu
paper_authors._id: 6552e2c9812ed6f6ea5945a2
paper_authors.name: Chunyi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T09:51:26.122Z
paper_authors._id: 644cf72f6d84e25fad73ae23
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644cf72f6d84e25fad73ae23/VF0V38obLNyKAbbTM65r8.jpeg
paper_authors.fullname: Chunyi Li
paper_authors.type: user
paper_authors.user: lcysyzxdxc
paper_authors._id: 6552e2c9812ed6f6ea5945a3
paper_authors.name: Jingwen Hou
paper_authors._id: 6552e2c9812ed6f6ea5945a4
paper_authors.name: Guangtao Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T12:32:05.837Z
paper_authors._id: 65535b125413c1a54e6fb243
paper_authors.avatarUrl: /avatars/03bcf1d58865f5406aff49a415e78bdc.svg
paper_authors.fullname: Guangtao Zhai
paper_authors.type: user
paper_authors.user: GTZhai
paper_authors._id: 6552e2c9812ed6f6ea5945a5
paper_authors.name: Geng Xue
paper_authors._id: 6552e2c9812ed6f6ea5945a6
paper_authors.name: Wenxiu Sun
paper_authors._id: 6552e2c9812ed6f6ea5945a7
paper_authors.name: Qiong Yan
paper_authors._id: 6552e2c9812ed6f6ea5945a8
paper_authors.name: Weisi Lin
paper_title: Q-Instruct: Improving Low-level Visual Abilities for Multi-modality
    Foundation Models
paper_summary: Multi-modality foundation models, as represented by GPT-4V, have brought a
  new paradigm for low-level visual perception and understanding tasks, that can
  respond to a broad range of natural human instructions in a model. While
  existing foundation models have shown exciting potentials on low-level visual
  tasks, their related abilities are still preliminary and need to be improved.
  In order to enhance these models, we conduct a large-scale subjective
  experiment collecting a vast number of real human feedbacks on low-level
  vision. Each feedback follows a pathway that starts with a detailed description
  on the low-level visual appearance (*e.g. clarity, color, brightness* of an
  image, and ends with an overall conclusion, with an average length of 45 words.
  The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
  18,973 images with diverse low-level appearance. Moreover, to enable foundation
  models to robustly respond to diverse types of questions, we design a
  GPT-participated conversion to process these feedbacks into diverse-format 200K
  instruction-response pairs. Experimental results indicate that the
  **Q-Instruct** consistently elevates low-level perception and understanding
  abilities across several foundational models. We anticipate that our datasets
  can pave the way for a future that general intelligence can perceive,
  understand low-level visual appearance and evaluate visual quality like a
  human. Our dataset, model zoo, and demo is published at:
  https://q-future.github.io/Q-Instruct.
paper_ai_keywords: Q-Pathway Q-Instruct