title: Leveraging Unpaired Data for Vision-Language Generative Models via Cycle
    Consistency
summary: Current vision-language generative models rely on expansive corpora of paired
  image-text data to attain optimal performance and generalization capabilities.
  However, automatically collecting such data (e.g. via large-scale web scraping)
  leads to low quality and poor image-text correlation, while human annotation is
  more accurate but requires significant manual effort and expense. We introduce
  ITIT (InTegrating Image
  Text): an innovative training paradigm grounded in the concept of
  cycle consistency which allows vision-language training on unpaired image and
  text data. ITIT is comprised of a joint image-text encoder with disjoint image
  and text decoders that enable bidirectional image-to-text and text-to-image
  generation in a single framework. During training, ITIT leverages a small set
  of paired image-text data to ensure its output matches the input reasonably
  well in both directions. Simultaneously, the model is also trained on much
  larger datasets containing only images or texts. This is achieved by enforcing
  cycle consistency between the original unpaired samples and the cycle-generated
  counterparts. For instance, it generates a caption for a given input image and
  then uses the caption to create an output image, and enforces similarity
  between the input and output images. Our experiments show that ITIT with
  unpaired datasets exhibits similar scaling behavior as using high-quality
  paired data. We demonstrate image generation and captioning performance on par
  with state-of-the-art text-to-image and image-to-text models with orders of
  magnitude fewer (only 3M) paired image-text data.
paper_authors._id: 651f6664063e54a50cb5f182
paper_authors.name: Tianhong Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-09T18:18:56.166Z
paper_authors._id: 65244352e0823fbfef8c6475
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65244352e0823fbfef8c6475/d6YipDIYHdhP_B6RJACNc.jpeg
paper_authors.fullname: Tianhong Li
paper_authors.type: user
paper_authors.user: LTH14
paper_authors._id: 651f6664063e54a50cb5f183
paper_authors.name: Sangnie Bhardwaj
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-11T07:34:46.710Z
paper_authors._id: 6525a62969946cd5aa21717a
paper_authors.avatarUrl: /avatars/81d382110a427b5541455ff947000a7f.svg
paper_authors.fullname: Sangnie Bhardwaj
paper_authors.type: user
paper_authors.user: sangnie
paper_authors._id: 651f6664063e54a50cb5f184
paper_authors.name: Yonglong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:52:23.092Z
paper_authors._id: 64793086ac77bb7603842456
paper_authors.avatarUrl: /avatars/5891292138d372a9cc063f86c08b8174.svg
paper_authors.fullname: Yonglong Tian
paper_authors.type: user
paper_authors.user: tyl5566
paper_authors._id: 651f6664063e54a50cb5f185
paper_authors.name: Han Zhang
paper_authors._id: 651f6664063e54a50cb5f186
paper_authors.name: Jarred Barber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:52:44.963Z
paper_authors._id: 63136d5154e6e5d9f0f7643b
paper_authors.avatarUrl: /avatars/6f3c8a0539da5964ca7e494178bd4f09.svg
paper_authors.fullname: Jarred
paper_authors.type: user
paper_authors.user: jarredbarber
paper_authors._id: 651f6664063e54a50cb5f187
paper_authors.name: Dina Katabi
paper_authors._id: 651f6664063e54a50cb5f188
paper_authors.name: Guillaume Lajoie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:52:59.050Z
paper_authors._id: 64909d1d6f217a26d91c6d9b
paper_authors.avatarUrl: /avatars/28eeff49dd5c2f01b576e4a050551e17.svg
paper_authors.fullname: guillaume lajoie
paper_authors.type: user
paper_authors.user: glajoie
paper_authors._id: 651f6664063e54a50cb5f189
paper_authors.name: Huiwen Chang
paper_authors._id: 651f6664063e54a50cb5f18a
paper_authors.name: Dilip Krishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:53:17.295Z
paper_authors._id: 6501603c12c1442d99618aa5
paper_authors.avatarUrl: /avatars/9c3d04da7fdb8b13b00b4486b75c9788.svg
paper_authors.fullname: Dilip Krishnan
paper_authors.type: user
paper_authors.user: dilipk8
paper_title: Leveraging Unpaired Data for Vision-Language Generative Models via Cycle
    Consistency
paper_summary: Current vision-language generative models rely on expansive corpora of paired
  image-text data to attain optimal performance and generalization capabilities.
  However, automatically collecting such data (e.g. via large-scale web scraping)
  leads to low quality and poor image-text correlation, while human annotation is
  more accurate but requires significant manual effort and expense. We introduce
  ITIT (InTegrating Image
  Text): an innovative training paradigm grounded in the concept of
  cycle consistency which allows vision-language training on unpaired image and
  text data. ITIT is comprised of a joint image-text encoder with disjoint image
  and text decoders that enable bidirectional image-to-text and text-to-image
  generation in a single framework. During training, ITIT leverages a small set
  of paired image-text data to ensure its output matches the input reasonably
  well in both directions. Simultaneously, the model is also trained on much
  larger datasets containing only images or texts. This is achieved by enforcing
  cycle consistency between the original unpaired samples and the cycle-generated
  counterparts. For instance, it generates a caption for a given input image and
  then uses the caption to create an output image, and enforces similarity
  between the input and output images. Our experiments show that ITIT with
  unpaired datasets exhibits similar scaling behavior as using high-quality
  paired data. We demonstrate image generation and captioning performance on par
  with state-of-the-art text-to-image and image-to-text models with orders of
  magnitude fewer (only 3M) paired image-text data.
paper_ai_summary: The ITIT model uses cycle consistency to train vision-language generative models on unpaired data, achieving performance comparable to those trained on large labeled datasets with significantly less paired data.
paper_ai_keywords: cycle consistency joint image-text encoder disjoint image and text decoders bidirectional image-to-text and text-to-image generation unpaired datasets similarity enforcement image generation captioning performance