Synth^2: Boosting Visual-Language Models with Synthetic Captions and
  Image Embeddings
The creation of high-quality human-labeled image-caption datasets presents a
significant bottleneck in the development of Visual-Language Models (VLMs). We
propose a novel approach that leverages the strengths of Large Language Models
(LLMs) and image generation models to create synthetic image-text pairs for
efficient and effective VLM training. Our method employs pretraining a
text-to-image model to synthesize image embeddings starting from captions
generated by an LLM. These synthetic pairs are then used to train a VLM.
Extensive experiments demonstrate that the VLM trained with synthetic data
exhibits comparable performance on image captioning, while requiring a fraction
of the data used by models trained solely on human-annotated data. In
particular, we outperform the baseline by 17% through augmentation with a
synthetic dataset. Furthermore, we show that synthesizing in the image
embedding space is 25% faster than in the pixel space. This research introduces
a promising technique for generating large-scale, customizable image datasets,
leading to enhanced VLM performance and wider applicability across various
domains, all with improved data efficiency and resource utilization.
65f11403555caedb29aae08f
Sahand Sharifzadeh
extracted_pending
2024-03-13T15:00:47.555Z
65f1bf93decdd4c67f489e3e
/avatars/dfca3c22752c9bec8f55a20b941fe6b4.svg
Sahand Sharifzadeh
user
sharifzadeh
65f11403555caedb29aae090
Christos Kaplanis
65f11403555caedb29aae091
Shreya Pathak
65f11403555caedb29aae092
Dharshan Kumaran
65f11403555caedb29aae093
Anastasija Ilic
65f11403555caedb29aae094
Jovana Mitrovic
65f11403555caedb29aae095
Charles Blundell
65f11403555caedb29aae096
Andrea Banino
claimed_verified
2024-03-13T13:52:12.461Z
65f1af6f75a7bb009dfa6301
/avatars/6f4d009e1faf9f67a6eb2f340b937212.svg
Andrea
user
abanino
Synth^2: Boosting Visual-Language Models with Synthetic Captions and
  Image Embeddings
The creation of high-quality human-labeled image-caption datasets presents a
significant bottleneck in the development of Visual-Language Models (VLMs). We
propose a novel approach that leverages the strengths of Large Language Models
(LLMs) and image generation models to create synthetic image-text pairs for
efficient and effective VLM training. Our method employs pretraining a
text-to-image model to synthesize image embeddings starting from captions
generated by an LLM. These synthetic pairs are then used to train a VLM.
Extensive experiments demonstrate that the VLM trained with synthetic data
exhibits comparable performance on image captioning, while requiring a fraction
of the data used by models trained solely on human-annotated data. In
particular, we outperform the baseline by 17% through augmentation with a
synthetic dataset. Furthermore, we show that synthesizing in the image
embedding space is 25% faster than in the pixel space. This research introduces
a promising technique for generating large-scale, customizable image datasets,
leading to enhanced VLM performance and wider applicability across various
domains, all with improved data efficiency and resource utilization.
A novel method using Large Language Models and text-to-image models generates synthetic image-text pairs, enhancing Visual-Language Model performance with less human-labeled data and improved efficiency.
Visual-Language Models (VLMs) Large Language Models (LLMs) text-to-image model image embeddings synthetic dataset image captioning pixel space