title: DiffusionGPT: LLM-Driven Text-to-Image Generation System
summary: Diffusion models have opened up new avenues for the field of image
  generation, resulting in the proliferation of high-quality models shared on
  open-source platforms. However, a major challenge persists in current
  text-to-image systems are often unable to handle diverse inputs, or are limited
  to single model results. Current unified attempts often fall into two
  orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
  expert model to output. To combine the best of both worlds, we propose
  DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
  generation system capable of seamlessly accommodating various types of prompts
  and integrating domain-expert models. DiffusionGPT constructs domain-specific
  Trees for various generative models based on prior knowledge. When provided
  with an input, the LLM parses the prompt and employs the Trees-of-Thought to
  guide the selection of an appropriate model, thereby relaxing input constraints
  and ensuring exceptional performance across diverse domains. Moreover, we
  introduce Advantage Databases, where the Tree-of-Thought is enriched with human
  feedback, aligning the model selection process with human preferences. Through
  extensive experiments and comparisons, we demonstrate the effectiveness of
  DiffusionGPT, showcasing its potential for pushing the boundaries of image
  synthesis in diverse domains.
paper_authors._id: 65a9e21dd766c6c6409f33eb
paper_authors.name: Jie Qin
paper_authors._id: 65a9e21dd766c6c6409f33ec
paper_authors.name: Jie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:31:32.405Z
paper_authors._id: 6381c5d63680a7cf34e08ca9
paper_authors.avatarUrl: /avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
paper_authors.fullname: wujie10558@gmail.com
paper_authors.type: user
paper_authors.user: wujie10
paper_authors._id: 65a9e21dd766c6c6409f33ed
paper_authors.name: Weifeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T07:42:21.798Z
paper_authors._id: 62a7fc03ac97233f161c2433
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7fc03ac97233f161c2433/CO3JDQVBZCnQxsgmnQHc1.jpeg
paper_authors.fullname: weifeng
paper_authors.type: user
paper_authors.user: weifeng-chen
paper_authors._id: 65a9e21dd766c6c6409f33ee
paper_authors.name: Yuxi Ren
paper_authors._id: 65a9e21dd766c6c6409f33ef
paper_authors.name: Huixia Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T11:19:19.974Z
paper_authors._id: 6368b457e943416fc80619fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6368b457e943416fc80619fb/RjAJKMwl6aIFKAZs1vgvK.jpeg
paper_authors.fullname: HXLee
paper_authors.type: user
paper_authors.user: HXLee
paper_authors._id: 65a9e21dd766c6c6409f33f0
paper_authors.name: Hefeng Wu
paper_authors._id: 65a9e21dd766c6c6409f33f1
paper_authors.name: Xuefeng Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:15:02.171Z
paper_authors._id: 646b7f71df2609a541c1ab9f
paper_authors.avatarUrl: /avatars/48b82e5fd9b06f41ff825507c36816cd.svg
paper_authors.fullname: Xuefeng Xiao
paper_authors.type: user
paper_authors.user: xiaoxuefeng
paper_authors._id: 65a9e21dd766c6c6409f33f2
paper_authors.name: Rui Wang
paper_authors._id: 65a9e21dd766c6c6409f33f3
paper_authors.name: Shilei Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:14:53.634Z
paper_authors._id: 6501763d12c1442d9964f883
paper_authors.avatarUrl: /avatars/3e06bb6afdecc0ab8ac9a5cd215c0265.svg
paper_authors.fullname: Shileiwen
paper_authors.type: user
paper_authors.user: Shileiwen
paper_title: DiffusionGPT: LLM-Driven Text-to-Image Generation System
paper_summary: Diffusion models have opened up new avenues for the field of image
  generation, resulting in the proliferation of high-quality models shared on
  open-source platforms. However, a major challenge persists in current
  text-to-image systems are often unable to handle diverse inputs, or are limited
  to single model results. Current unified attempts often fall into two
  orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
  expert model to output. To combine the best of both worlds, we propose
  DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
  generation system capable of seamlessly accommodating various types of prompts
  and integrating domain-expert models. DiffusionGPT constructs domain-specific
  Trees for various generative models based on prior knowledge. When provided
  with an input, the LLM parses the prompt and employs the Trees-of-Thought to
  guide the selection of an appropriate model, thereby relaxing input constraints
  and ensuring exceptional performance across diverse domains. Moreover, we
  introduce Advantage Databases, where the Tree-of-Thought is enriched with human
  feedback, aligning the model selection process with human preferences. Through
  extensive experiments and comparisons, we demonstrate the effectiveness of
  DiffusionGPT, showcasing its potential for pushing the boundaries of image
  synthesis in diverse domains.
paper_ai_keywords: diffusion models text-to-image systems Large Language Models (LLM) DiffusionGPT Trees generative models Trees-of-Thought Advantage Databases