title: Synth^2: Boosting Visual-Language Models with Synthetic Captions and
    Image Embeddings
summary: The creation of high-quality human-labeled image-caption datasets presents a
  significant bottleneck in the development of Visual-Language Models (VLMs). We
  propose a novel approach that leverages the strengths of Large Language Models
  (LLMs) and image generation models to create synthetic image-text pairs for
  efficient and effective VLM training. Our method employs pretraining a
  text-to-image model to synthesize image embeddings starting from captions
  generated by an LLM. These synthetic pairs are then used to train a VLM.
  Extensive experiments demonstrate that the VLM trained with synthetic data
  exhibits comparable performance on image captioning, while requiring a fraction
  of the data used by models trained solely on human-annotated data. In
  particular, we outperform the baseline by 17% through augmentation with a
  synthetic dataset. Furthermore, we show that synthesizing in the image
  embedding space is 25% faster than in the pixel space. This research introduces
  a promising technique for generating large-scale, customizable image datasets,
  leading to enhanced VLM performance and wider applicability across various
  domains, all with improved data efficiency and resource utilization.
paper_authors._id: 65f11403555caedb29aae08f
paper_authors.name: Sahand Sharifzadeh
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-13T15:00:47.555Z
paper_authors._id: 65f1bf93decdd4c67f489e3e
paper_authors.avatarUrl: /avatars/dfca3c22752c9bec8f55a20b941fe6b4.svg
paper_authors.fullname: Sahand Sharifzadeh
paper_authors.type: user
paper_authors.user: sharifzadeh
paper_authors._id: 65f11403555caedb29aae090
paper_authors.name: Christos Kaplanis
paper_authors._id: 65f11403555caedb29aae091
paper_authors.name: Shreya Pathak
paper_authors._id: 65f11403555caedb29aae092
paper_authors.name: Dharshan Kumaran
paper_authors._id: 65f11403555caedb29aae093
paper_authors.name: Anastasija Ilic
paper_authors._id: 65f11403555caedb29aae094
paper_authors.name: Jovana Mitrovic
paper_authors._id: 65f11403555caedb29aae095
paper_authors.name: Charles Blundell
paper_authors._id: 65f11403555caedb29aae096
paper_authors.name: Andrea Banino
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T13:52:12.461Z
paper_authors._id: 65f1af6f75a7bb009dfa6301
paper_authors.avatarUrl: /avatars/6f4d009e1faf9f67a6eb2f340b937212.svg
paper_authors.fullname: Andrea
paper_authors.type: user
paper_authors.user: abanino
paper_title: Synth^2: Boosting Visual-Language Models with Synthetic Captions and
    Image Embeddings
paper_summary: The creation of high-quality human-labeled image-caption datasets presents a
  significant bottleneck in the development of Visual-Language Models (VLMs). We
  propose a novel approach that leverages the strengths of Large Language Models
  (LLMs) and image generation models to create synthetic image-text pairs for
  efficient and effective VLM training. Our method employs pretraining a
  text-to-image model to synthesize image embeddings starting from captions
  generated by an LLM. These synthetic pairs are then used to train a VLM.
  Extensive experiments demonstrate that the VLM trained with synthetic data
  exhibits comparable performance on image captioning, while requiring a fraction
  of the data used by models trained solely on human-annotated data. In
  particular, we outperform the baseline by 17% through augmentation with a
  synthetic dataset. Furthermore, we show that synthesizing in the image
  embedding space is 25% faster than in the pixel space. This research introduces
  a promising technique for generating large-scale, customizable image datasets,
  leading to enhanced VLM performance and wider applicability across various
  domains, all with improved data efficiency and resource utilization.
paper_ai_summary: A novel method using Large Language Models and text-to-image models generates synthetic image-text pairs, enhancing Visual-Language Model performance with less human-labeled data and improved efficiency.
paper_ai_keywords: Visual-Language Models (VLMs) Large Language Models (LLMs) text-to-image model image embeddings synthetic dataset image captioning pixel space