title: Language-Informed Visual Concept Learning
summary: Our understanding of the visual world is centered around various concept
  axes, characterizing different aspects of visual entities. While different
  concept axes can be easily specified by language, e.g. color, the exact visual
  nuances along each axis often exceed the limitations of linguistic
  articulations, e.g. a particular style of painting. In this work, our goal is
  to learn a language-informed visual concept representation, by simply
  distilling large pre-trained vision-language models. Specifically, we train a
  set of concept encoders to encode the information pertinent to a set of
  language-informed concept axes, with an objective of reproducing the input
  image through a pre-trained Text-to-Image (T2I) model. To encourage better
  disentanglement of different concept encoders, we anchor the concept embeddings
  to a set of text embeddings obtained from a pre-trained Visual Question
  Answering (VQA) model. At inference time, the model extracts concept embeddings
  along various axes from new test images, which can be remixed to generate
  images with novel compositions of visual concepts. With a lightweight test-time
  finetuning procedure, it can also generalize to novel concepts unseen at
  training.
paper_authors._id: 657126d6971de7383ef87398
paper_authors.name: Sharon Lee
paper_authors._id: 657126d6971de7383ef87399
paper_authors.name: Yunzhi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:25:55.589Z
paper_authors._id: 63793aee2cb50dda39d6e73b
paper_authors.avatarUrl: /avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
paper_authors.fullname: Yunzhi Zhang
paper_authors.type: user
paper_authors.user: Yunzhi
paper_authors._id: 657126d6971de7383ef8739a
paper_authors.name: Shangzhe Wu
paper_authors._id: 657126d6971de7383ef8739b
paper_authors.name: Jiajun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-21T09:31:58.926Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_title: Language-Informed Visual Concept Learning
paper_summary: Our understanding of the visual world is centered around various concept
  axes, characterizing different aspects of visual entities. While different
  concept axes can be easily specified by language, e.g. color, the exact visual
  nuances along each axis often exceed the limitations of linguistic
  articulations, e.g. a particular style of painting. In this work, our goal is
  to learn a language-informed visual concept representation, by simply
  distilling large pre-trained vision-language models. Specifically, we train a
  set of concept encoders to encode the information pertinent to a set of
  language-informed concept axes, with an objective of reproducing the input
  image through a pre-trained Text-to-Image (T2I) model. To encourage better
  disentanglement of different concept encoders, we anchor the concept embeddings
  to a set of text embeddings obtained from a pre-trained Visual Question
  Answering (VQA) model. At inference time, the model extracts concept embeddings
  along various axes from new test images, which can be remixed to generate
  images with novel compositions of visual concepts. With a lightweight test-time
  finetuning procedure, it can also generalize to novel concepts unseen at
  training.
paper_ai_summary: A language-informed visual concept representation is learned by distilling pre-trained vision-language models, enabling disentangled concept encodings and novel image generation through remixing.
paper_ai_keywords: concept encoders Text-to-Image (T2I) model concept embeddings Visual Question Answering (VQA) model lightweight test-time finetuning