Language-Informed Visual Concept Learning
Our understanding of the visual world is centered around various concept
axes, characterizing different aspects of visual entities. While different
concept axes can be easily specified by language, e.g. color, the exact visual
nuances along each axis often exceed the limitations of linguistic
articulations, e.g. a particular style of painting. In this work, our goal is
to learn a language-informed visual concept representation, by simply
distilling large pre-trained vision-language models. Specifically, we train a
set of concept encoders to encode the information pertinent to a set of
language-informed concept axes, with an objective of reproducing the input
image through a pre-trained Text-to-Image (T2I) model. To encourage better
disentanglement of different concept encoders, we anchor the concept embeddings
to a set of text embeddings obtained from a pre-trained Visual Question
Answering (VQA) model. At inference time, the model extracts concept embeddings
along various axes from new test images, which can be remixed to generate
images with novel compositions of visual concepts. With a lightweight test-time
finetuning procedure, it can also generalize to novel concepts unseen at
training.
657126d6971de7383ef87398
Sharon Lee
657126d6971de7383ef87399
Yunzhi Zhang
admin_assigned
2023-12-07T12:25:55.589Z
63793aee2cb50dda39d6e73b
/avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
Yunzhi Zhang
user
Yunzhi
657126d6971de7383ef8739a
Shangzhe Wu
657126d6971de7383ef8739b
Jiajun Wu
claimed_verified
2023-12-21T09:31:58.926Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
Language-Informed Visual Concept Learning
Our understanding of the visual world is centered around various concept
axes, characterizing different aspects of visual entities. While different
concept axes can be easily specified by language, e.g. color, the exact visual
nuances along each axis often exceed the limitations of linguistic
articulations, e.g. a particular style of painting. In this work, our goal is
to learn a language-informed visual concept representation, by simply
distilling large pre-trained vision-language models. Specifically, we train a
set of concept encoders to encode the information pertinent to a set of
language-informed concept axes, with an objective of reproducing the input
image through a pre-trained Text-to-Image (T2I) model. To encourage better
disentanglement of different concept encoders, we anchor the concept embeddings
to a set of text embeddings obtained from a pre-trained Visual Question
Answering (VQA) model. At inference time, the model extracts concept embeddings
along various axes from new test images, which can be remixed to generate
images with novel compositions of visual concepts. With a lightweight test-time
finetuning procedure, it can also generalize to novel concepts unseen at
training.
A language-informed visual concept representation is learned by distilling pre-trained vision-language models, enabling disentangled concept encodings and novel image generation through remixing.
concept encoders Text-to-Image (T2I) model concept embeddings Visual Question Answering (VQA) model lightweight test-time finetuning