title: MAGID: An Automated Pipeline for Generating Synthetic Multi-modal
    Datasets
summary: Development of multimodal interactive systems is hindered by the lack of
  rich, multimodal (text, images) conversational data, which is needed in large
  quantities for LLMs. Previous approaches augment textual dialogues with
  retrieved images, posing privacy, diversity, and quality constraints. In this
  work, we introduce Multimodal Augmented Generative
  Images Dialogues (MAGID), a framework to augment text-only
  dialogues with diverse and high-quality images. Subsequently, a diffusion model
  is applied to craft corresponding images, ensuring alignment with the
  identified text. Finally, MAGID incorporates an innovative feedback loop
  between an image description generation module (textual LLM) and image quality
  modules (addressing aesthetics, image-text matching, and safety), that work in
  tandem to generate high-quality and multi-modal dialogues. We compare MAGID to
  other SOTA baselines on three dialogue datasets, using automated and human
  evaluation. Our results show that MAGID is comparable to or better than
  baselines, with significant improvements in human evaluation, especially
  against retrieval baselines where the image database is small.
paper_authors._id: 65e7f7b0b87e5297e4718247
paper_authors.name: Hossein Aboutalebi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:08:17.281Z
paper_authors._id: 6470ed00d9360cd9d8e8ff87
paper_authors.avatarUrl: /avatars/17f904a9ff8b87d83e279c479a3547a6.svg
paper_authors.fullname: Hossein Aboutalebi
paper_authors.type: user
paper_authors.user: SeattleIsBest
paper_authors._id: 65e7f7b0b87e5297e4718248
paper_authors.name: Hwanjun Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:08:28.059Z
paper_authors._id: 63c9da8d5fdc575773c84816
paper_authors.avatarUrl: /avatars/9357452f7df22c98a3ac6dbb90aa1b14.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Hwanjun
paper_authors._id: 65e7f7b0b87e5297e4718249
paper_authors.name: Yusheng Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:08:33.949Z
paper_authors._id: 627b3008f5ef71071a3f7f51
paper_authors.avatarUrl: /avatars/0e17d4f424d67a5983f2915d52c26860.svg
paper_authors.fullname: Yusheng Xie
paper_authors.type: user
paper_authors.user: yushx
paper_authors._id: 65e7f7b0b87e5297e471824a
paper_authors.name: Arshit Gupta
paper_authors._id: 65e7f7b0b87e5297e471824b
paper_authors.name: Justin Sun
paper_authors._id: 65e7f7b0b87e5297e471824c
paper_authors.name: Hang Su
paper_authors._id: 65e7f7b0b87e5297e471824d
paper_authors.name: Igor Shalyminov
paper_authors._id: 65e7f7b0b87e5297e471824e
paper_authors.name: Nikolaos Pappas
paper_authors._id: 65e7f7b0b87e5297e471824f
paper_authors.name: Siffi Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:09:57.405Z
paper_authors._id: 63988cf124e61e0d386f22a3
paper_authors.avatarUrl: /avatars/a39d485f8a0ead0e74ad6d5a612e7908.svg
paper_authors.fullname: Siffi Singh
paper_authors.type: user
paper_authors.user: siffi26
paper_authors._id: 65e7f7b0b87e5297e4718250
paper_authors.name: Saab Mansour
paper_title: MAGID: An Automated Pipeline for Generating Synthetic Multi-modal
    Datasets
paper_summary: Development of multimodal interactive systems is hindered by the lack of
  rich, multimodal (text, images) conversational data, which is needed in large
  quantities for LLMs. Previous approaches augment textual dialogues with
  retrieved images, posing privacy, diversity, and quality constraints. In this
  work, we introduce Multimodal Augmented Generative
  Images Dialogues (MAGID), a framework to augment text-only
  dialogues with diverse and high-quality images. Subsequently, a diffusion model
  is applied to craft corresponding images, ensuring alignment with the
  identified text. Finally, MAGID incorporates an innovative feedback loop
  between an image description generation module (textual LLM) and image quality
  modules (addressing aesthetics, image-text matching, and safety), that work in
  tandem to generate high-quality and multi-modal dialogues. We compare MAGID to
  other SOTA baselines on three dialogue datasets, using automated and human
  evaluation. Our results show that MAGID is comparable to or better than
  baselines, with significant improvements in human evaluation, especially
  against retrieval baselines where the image database is small.
paper_ai_summary: MAGID, a framework augmenting text-only dialogues with high-quality images using a diffusion model and feedback loops, outperforms existing multimodal dialogue generation methods.
paper_ai_keywords: diffusion model image-text matching multimodal dialogues image quality feedback loop textual LLM aesthetic assessment safety assessment