title: Silkie: Preference Distillation for Large Visual Language Models
summary: This paper explores preference distillation for large vision language models
  (LVLMs), improving their ability to generate helpful and faithful responses
  anchoring the visual context. We first build a vision-language feedback
  (VLFeedback) dataset utilizing AI annotation. Specifically, responses are
  generated by models sampled from 12 LVLMs, conditioned on multi-modal
  instructions sourced from various datasets. We adopt GPT-4V to assess the
  generated outputs regarding helpfulness, visual faithfulness, and ethical
  considerations. Furthermore, the preference supervision is distilled into
  Qwen-VL-Chat through the direct preference optimization (DPO) method. The
  resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
  benchmark regarding the perception and cognition capabilities, respectively.
  Silkie also demonstrates reduced hallucination by setting a new
  state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis
  shows that DPO with our VLFeedback dataset mainly boosts the fine-grained
  perception and complex cognition abilities of LVLMs, leading to more
  comprehensive improvements compared to human-annotated preference datasets.
paper_authors._id: 65812a0ee8c8d33e565205e5
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T08:19:59.505Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 65812a0ee8c8d33e565205e6
paper_authors.name: Zhihui Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T14:23:52.699Z
paper_authors._id: 622f103fc78da4c7ebd7c887
paper_authors.avatarUrl: /avatars/b0c7cd29835d92c2cd584947fcd5d520.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Zhihui
paper_authors._id: 65812a0ee8c8d33e565205e7
paper_authors.name: Mukai Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-19T08:31:38.150Z
paper_authors._id: 624561c1939c9acec2103534
paper_authors.avatarUrl: /avatars/3efdf797c53d153cea7415213fb5afc7.svg
paper_authors.fullname: Mukai Li
paper_authors.type: user
paper_authors.user: kiaia
paper_authors._id: 65812a0ee8c8d33e565205e8
paper_authors.name: Shunian Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T14:24:08.180Z
paper_authors._id: 623be9e1d1eb227788764959
paper_authors.avatarUrl: /avatars/b6521b795a59754dbb40123fd4f63b8c.svg
paper_authors.fullname: Shunian Chen
paper_authors.type: user
paper_authors.user: Shunian
paper_authors._id: 65812a0ee8c8d33e565205e9
paper_authors.name: Peiyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T14:24:16.670Z
paper_authors._id: 656873f33fd0bf1f82558695
paper_authors.avatarUrl: /avatars/7a085da2e2a91d7f41988501a573ebf9.svg
paper_authors.fullname: PEIYI, WANG
paper_authors.type: user
paper_authors.user: peiyiwang89
paper_authors._id: 65812a0ee8c8d33e565205ea
paper_authors.name: Liang Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-12T06:27:39.228Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 65812a0ee8c8d33e565205eb
paper_authors.name: Yazheng Yang
paper_authors._id: 65812a0ee8c8d33e565205ec
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T14:24:40.275Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 65812a0ee8c8d33e565205ed
paper_authors.name: Lingpeng Kong
paper_title: Silkie: Preference Distillation for Large Visual Language Models
paper_summary: This paper explores preference distillation for large vision language models
  (LVLMs), improving their ability to generate helpful and faithful responses
  anchoring the visual context. We first build a vision-language feedback
  (VLFeedback) dataset utilizing AI annotation. Specifically, responses are
  generated by models sampled from 12 LVLMs, conditioned on multi-modal
  instructions sourced from various datasets. We adopt GPT-4V to assess the
  generated outputs regarding helpfulness, visual faithfulness, and ethical
  considerations. Furthermore, the preference supervision is distilled into
  Qwen-VL-Chat through the direct preference optimization (DPO) method. The
  resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
  benchmark regarding the perception and cognition capabilities, respectively.
  Silkie also demonstrates reduced hallucination by setting a new
  state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis
  shows that DPO with our VLFeedback dataset mainly boosts the fine-grained
  perception and complex cognition abilities of LVLMs, leading to more
  comprehensive improvements compared to human-annotated preference datasets.
paper_ai_keywords: preference distillation vision language models (LVLMs) VLFeedback dataset multi-modal instructions GPT-4V helpfulness visual fidelity ethical considerations preference supervision Qwen-VL-Chat direct preference optimization (DPO) Silkie MME benchmark perception capabilities cognition capabilities hallucination MMHal-Bench benchmark fine-grained perception complex cognition abilities