Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Instruction following vision-language (VL) models offer a flexible interface
that supports a broad range of multimodal tasks in a zero-shot fashion.
However, interfaces that operate on full images do not directly enable the user
to "point to" and access specific regions within images. This capability is
important not only to support reference-grounded VL benchmarks, but also, for
practical applications that require precise within-image reasoning. We build
Localized Visual Commonsense models, which allow users to specify (multiple)
regions as input. We train our model by sampling localized commonsense
knowledge from a large language model (LLM): specifically, we prompt an LLM to
collect commonsense knowledge given a global literal image description and a
local literal region description automatically generated by a set of VL models.
With a separately trained critic model that selects high-quality examples, we
find that training on the localized commonsense corpus can successfully distill
existing VL models to support a reference-as-input interface. Empirical results
and human evaluations in a zero-shot setup demonstrate that our distillation
method results in more precise VL models of reasoning compared to a baseline of
passing a generated referring expression to an LLM.
65767508015c459f1be2c6c4
Jae Sung Park
admin_assigned
2023-12-11T08:53:03.049Z
6345a11743f4f2d2ed1057ca
/avatars/cba8b194b2517da76021d8cc1814a542.svg
Jae Sung Park
user
jamepark3922
65767508015c459f1be2c6c5
Jack Hessel
admin_assigned
2023-12-11T08:53:09.318Z
625498644f4edf771516b2cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
Jack Hessel
user
jmhessel
65767508015c459f1be2c6c6
Khyathi Raghavi Chandu
admin_assigned
2023-12-11T08:53:15.708Z
62ebe6472f8999993a847bb6
/avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
Khyathi Raghavi Chandu
user
khyathi
65767508015c459f1be2c6c7
Paul Pu Liang
65767508015c459f1be2c6c8
Ximing Lu
admin_assigned
2023-12-11T08:53:29.716Z
640928bd3461c51cf7378707
/avatars/b29fcb7388b81f8686086352f6321d06.svg
Ximing Lu
user
Ximing
65767508015c459f1be2c6c9
Peter West
admin_assigned
2023-12-11T08:53:49.701Z
6359e982548e5c7e6e6b2154
/avatars/0cda044c09ca95254e001eebd00aebe8.svg
Peter West
user
peterwestuw
65767508015c459f1be2c6ca
Youngjae Yu
65767508015c459f1be2c6cb
Qiuyuan Huang
65767508015c459f1be2c6cc
Jianfeng Gao
admin_assigned
2023-12-11T08:54:39.004Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
65767508015c459f1be2c6cd
Ali Farhadi
65767508015c459f1be2c6ce
Yejin Choi
admin_assigned
2023-12-11T08:54:49.644Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Instruction following vision-language (VL) models offer a flexible interface
that supports a broad range of multimodal tasks in a zero-shot fashion.
However, interfaces that operate on full images do not directly enable the user
to "point to" and access specific regions within images. This capability is
important not only to support reference-grounded VL benchmarks, but also, for
practical applications that require precise within-image reasoning. We build
Localized Visual Commonsense models, which allow users to specify (multiple)
regions as input. We train our model by sampling localized commonsense
knowledge from a large language model (LLM): specifically, we prompt an LLM to
collect commonsense knowledge given a global literal image description and a
local literal region description automatically generated by a set of VL models.
With a separately trained critic model that selects high-quality examples, we
find that training on the localized commonsense corpus can successfully distill
existing VL models to support a reference-as-input interface. Empirical results
and human evaluations in a zero-shot setup demonstrate that our distillation
method results in more precise VL models of reasoning compared to a baseline of
passing a generated referring expression to an LLM.
Localized Visual Commonsense models are trained to reason about specific regions in images using commonsense knowledge extracted from a language model, leading to more precise zero-shot performance compared to models that use referring expressions.
vision-language (VL) models multimodal tasks zero-shot fashion localized commonsense knowledge large language model (LLM) global literal image description local literal region description critic model distillation method