MAGID: An Automated Pipeline for Generating Synthetic Multi-modal
  Datasets
Development of multimodal interactive systems is hindered by the lack of
rich, multimodal (text, images) conversational data, which is needed in large
quantities for LLMs. Previous approaches augment textual dialogues with
retrieved images, posing privacy, diversity, and quality constraints. In this
work, we introduce Multimodal Augmented Generative
Images Dialogues (MAGID), a framework to augment text-only
dialogues with diverse and high-quality images. Subsequently, a diffusion model
is applied to craft corresponding images, ensuring alignment with the
identified text. Finally, MAGID incorporates an innovative feedback loop
between an image description generation module (textual LLM) and image quality
modules (addressing aesthetics, image-text matching, and safety), that work in
tandem to generate high-quality and multi-modal dialogues. We compare MAGID to
other SOTA baselines on three dialogue datasets, using automated and human
evaluation. Our results show that MAGID is comparable to or better than
baselines, with significant improvements in human evaluation, especially
against retrieval baselines where the image database is small.
65e7f7b0b87e5297e4718247
Hossein Aboutalebi
admin_assigned
2024-03-06T10:08:17.281Z
6470ed00d9360cd9d8e8ff87
/avatars/17f904a9ff8b87d83e279c479a3547a6.svg
Hossein Aboutalebi
user
SeattleIsBest
65e7f7b0b87e5297e4718248
Hwanjun Song
admin_assigned
2024-03-06T10:08:28.059Z
63c9da8d5fdc575773c84816
/avatars/9357452f7df22c98a3ac6dbb90aa1b14.svg
Song
user
Hwanjun
65e7f7b0b87e5297e4718249
Yusheng Xie
admin_assigned
2024-03-06T10:08:33.949Z
627b3008f5ef71071a3f7f51
/avatars/0e17d4f424d67a5983f2915d52c26860.svg
Yusheng Xie
user
yushx
65e7f7b0b87e5297e471824a
Arshit Gupta
65e7f7b0b87e5297e471824b
Justin Sun
65e7f7b0b87e5297e471824c
Hang Su
65e7f7b0b87e5297e471824d
Igor Shalyminov
65e7f7b0b87e5297e471824e
Nikolaos Pappas
65e7f7b0b87e5297e471824f
Siffi Singh
admin_assigned
2024-03-06T10:09:57.405Z
63988cf124e61e0d386f22a3
/avatars/a39d485f8a0ead0e74ad6d5a612e7908.svg
Siffi Singh
user
siffi26
65e7f7b0b87e5297e4718250
Saab Mansour
MAGID: An Automated Pipeline for Generating Synthetic Multi-modal
  Datasets
Development of multimodal interactive systems is hindered by the lack of
rich, multimodal (text, images) conversational data, which is needed in large
quantities for LLMs. Previous approaches augment textual dialogues with
retrieved images, posing privacy, diversity, and quality constraints. In this
work, we introduce Multimodal Augmented Generative
Images Dialogues (MAGID), a framework to augment text-only
dialogues with diverse and high-quality images. Subsequently, a diffusion model
is applied to craft corresponding images, ensuring alignment with the
identified text. Finally, MAGID incorporates an innovative feedback loop
between an image description generation module (textual LLM) and image quality
modules (addressing aesthetics, image-text matching, and safety), that work in
tandem to generate high-quality and multi-modal dialogues. We compare MAGID to
other SOTA baselines on three dialogue datasets, using automated and human
evaluation. Our results show that MAGID is comparable to or better than
baselines, with significant improvements in human evaluation, especially
against retrieval baselines where the image database is small.
MAGID, a framework augmenting text-only dialogues with high-quality images using a diffusion model and feedback loops, outperforms existing multimodal dialogue generation methods.
diffusion model image-text matching multimodal dialogues image quality feedback loop textual LLM aesthetic assessment safety assessment