title: Unified Text-to-Image Generation and Retrieval
summary: How humans can efficiently and effectively acquire images has always been a
  perennial question. A typical solution is text-to-image retrieval from an
  existing database given the text query; however, the limited database typically
  lacks creativity. By contrast, recent breakthroughs in text-to-image generation
  have made it possible to produce fancy and diverse visual content, but it faces
  challenges in synthesizing knowledge-intensive images. In this work, we rethink
  the relationship between text-to-image generation and retrieval and propose a
  unified framework in the context of Multimodal Large Language Models (MLLMs).
  Specifically, we first explore the intrinsic discriminative abilities of MLLMs
  and introduce a generative retrieval method to perform retrieval in a
  training-free manner. Subsequently, we unify generation and retrieval in an
  autoregressive generation way and propose an autonomous decision module to
  choose the best-matched one between generated and retrieved images as the
  response to the text query. Additionally, we construct a benchmark called
  TIGeR-Bench, including creative and knowledge-intensive domains, to standardize
  the evaluation of unified text-to-image generation and retrieval. Extensive
  experimental results on TIGeR-Bench and two retrieval benchmarks, i.e.,
  Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our
  proposed method.
paper_authors._id: 6667c486f3d613f1b1b2ef97
paper_authors.name: Leigang Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:09:40.697Z
paper_authors._id: 641a6e324182690729c860b5
paper_authors.avatarUrl: /avatars/4599451d9775bf1aa11ef1e7011cb05a.svg
paper_authors.fullname: Leigang Qu
paper_authors.type: user
paper_authors.user: leigangqu
paper_authors._id: 6667c486f3d613f1b1b2ef98
paper_authors.name: Haochuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T09:21:12.546Z
paper_authors._id: 64723663b628f9ecdf078fee
paper_authors.avatarUrl: /avatars/3367ea7edf1355d9c3e42da09f7dd021.svg
paper_authors.fullname: Haochuan Li
paper_authors.type: user
paper_authors.user: Haochuan
paper_authors._id: 6667c486f3d613f1b1b2ef99
paper_authors.name: Tan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:10:18.556Z
paper_authors._id: 62a9cb5c0036766ec1b67f09
paper_authors.avatarUrl: /avatars/5a9c5b41ef3eebbe7b21184b289ade68.svg
paper_authors.fullname: Tan Wang
paper_authors.type: user
paper_authors.user: ReaLMe
paper_authors._id: 6667c486f3d613f1b1b2ef9a
paper_authors.name: Wenjie Wang
paper_authors._id: 6667c486f3d613f1b1b2ef9b
paper_authors.name: Yongqi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:10:50.513Z
paper_authors._id: 639284bce3fa7a0619408feb
paper_authors.avatarUrl: /avatars/c5a019abb6858c35524dea5ab2036f26.svg
paper_authors.fullname: Yongqi Li
paper_authors.type: user
paper_authors.user: ylic
paper_authors._id: 6667c486f3d613f1b1b2ef9c
paper_authors.name: Liqiang Nie
paper_authors._id: 6667c486f3d613f1b1b2ef9d
paper_authors.name: Tat-Seng Chua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:11:04.163Z
paper_authors._id: 6570ae84c4993b8fb96f41a8
paper_authors.avatarUrl: /avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
paper_authors.fullname: Tat-Seng Chua
paper_authors.type: user
paper_authors.user: chuats
paper_title: Unified Text-to-Image Generation and Retrieval
paper_summary: How humans can efficiently and effectively acquire images has always been a
  perennial question. A typical solution is text-to-image retrieval from an
  existing database given the text query; however, the limited database typically
  lacks creativity. By contrast, recent breakthroughs in text-to-image generation
  have made it possible to produce fancy and diverse visual content, but it faces
  challenges in synthesizing knowledge-intensive images. In this work, we rethink
  the relationship between text-to-image generation and retrieval and propose a
  unified framework in the context of Multimodal Large Language Models (MLLMs).
  Specifically, we first explore the intrinsic discriminative abilities of MLLMs
  and introduce a generative retrieval method to perform retrieval in a
  training-free manner. Subsequently, we unify generation and retrieval in an
  autoregressive generation way and propose an autonomous decision module to
  choose the best-matched one between generated and retrieved images as the
  response to the text query. Additionally, we construct a benchmark called
  TIGeR-Bench, including creative and knowledge-intensive domains, to standardize
  the evaluation of unified text-to-image generation and retrieval. Extensive
  experimental results on TIGeR-Bench and two retrieval benchmarks, i.e.,
  Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our
  proposed method.
paper_ai_summary: A unified framework combining text-to-image generation and retrieval using Multimodal Large Language Models improves the quality and diversity of generated images, particularly for knowledge-intensive content.
paper_ai_keywords: Multimodal Large Language Models generative retrieval autoregressive generation autonomous decision module TIGeR-Bench Flickr30K MS-COCO