title: Localized Symbolic Knowledge Distillation for Visual Commonsense Models
summary: Instruction following vision-language (VL) models offer a flexible interface
  that supports a broad range of multimodal tasks in a zero-shot fashion.
  However, interfaces that operate on full images do not directly enable the user
  to "point to" and access specific regions within images. This capability is
  important not only to support reference-grounded VL benchmarks, but also, for
  practical applications that require precise within-image reasoning. We build
  Localized Visual Commonsense models, which allow users to specify (multiple)
  regions as input. We train our model by sampling localized commonsense
  knowledge from a large language model (LLM): specifically, we prompt an LLM to
  collect commonsense knowledge given a global literal image description and a
  local literal region description automatically generated by a set of VL models.
  With a separately trained critic model that selects high-quality examples, we
  find that training on the localized commonsense corpus can successfully distill
  existing VL models to support a reference-as-input interface. Empirical results
  and human evaluations in a zero-shot setup demonstrate that our distillation
  method results in more precise VL models of reasoning compared to a baseline of
  passing a generated referring expression to an LLM.
paper_authors._id: 65767508015c459f1be2c6c4
paper_authors.name: Jae Sung Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:53:03.049Z
paper_authors._id: 6345a11743f4f2d2ed1057ca
paper_authors.avatarUrl: /avatars/cba8b194b2517da76021d8cc1814a542.svg
paper_authors.fullname: Jae Sung Park
paper_authors.type: user
paper_authors.user: jamepark3922
paper_authors._id: 65767508015c459f1be2c6c5
paper_authors.name: Jack Hessel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:53:09.318Z
paper_authors._id: 625498644f4edf771516b2cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
paper_authors.fullname: Jack Hessel
paper_authors.type: user
paper_authors.user: jmhessel
paper_authors._id: 65767508015c459f1be2c6c6
paper_authors.name: Khyathi Raghavi Chandu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:53:15.708Z
paper_authors._id: 62ebe6472f8999993a847bb6
paper_authors.avatarUrl: /avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
paper_authors.fullname: Khyathi Raghavi Chandu
paper_authors.type: user
paper_authors.user: khyathi
paper_authors._id: 65767508015c459f1be2c6c7
paper_authors.name: Paul Pu Liang
paper_authors._id: 65767508015c459f1be2c6c8
paper_authors.name: Ximing Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:53:29.716Z
paper_authors._id: 640928bd3461c51cf7378707
paper_authors.avatarUrl: /avatars/429d1ba8327df82a8f8360c0f55e199c.svg
paper_authors.fullname: Ximing Lu
paper_authors.type: user
paper_authors.user: Ximing
paper_authors._id: 65767508015c459f1be2c6c9
paper_authors.name: Peter West
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:53:49.701Z
paper_authors._id: 6359e982548e5c7e6e6b2154
paper_authors.avatarUrl: /avatars/0cda044c09ca95254e001eebd00aebe8.svg
paper_authors.fullname: Peter West
paper_authors.type: user
paper_authors.user: peterwestuw
paper_authors._id: 65767508015c459f1be2c6ca
paper_authors.name: Youngjae Yu
paper_authors._id: 65767508015c459f1be2c6cb
paper_authors.name: Qiuyuan Huang
paper_authors._id: 65767508015c459f1be2c6cc
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:54:39.004Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 65767508015c459f1be2c6cd
paper_authors.name: Ali Farhadi
paper_authors._id: 65767508015c459f1be2c6ce
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:54:49.644Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_title: Localized Symbolic Knowledge Distillation for Visual Commonsense Models
paper_summary: Instruction following vision-language (VL) models offer a flexible interface
  that supports a broad range of multimodal tasks in a zero-shot fashion.
  However, interfaces that operate on full images do not directly enable the user
  to "point to" and access specific regions within images. This capability is
  important not only to support reference-grounded VL benchmarks, but also, for
  practical applications that require precise within-image reasoning. We build
  Localized Visual Commonsense models, which allow users to specify (multiple)
  regions as input. We train our model by sampling localized commonsense
  knowledge from a large language model (LLM): specifically, we prompt an LLM to
  collect commonsense knowledge given a global literal image description and a
  local literal region description automatically generated by a set of VL models.
  With a separately trained critic model that selects high-quality examples, we
  find that training on the localized commonsense corpus can successfully distill
  existing VL models to support a reference-as-input interface. Empirical results
  and human evaluations in a zero-shot setup demonstrate that our distillation
  method results in more precise VL models of reasoning compared to a baseline of
  passing a generated referring expression to an LLM.
paper_ai_keywords: vision-language (VL) models multimodal tasks zero-shot fashion localized commonsense models localized commonsense knowledge large language model (LLM) global literal image description local literal region description neural-quality examples reference-as-input interface distillation method referring expression