DiffusionGPT: LLM-Driven Text-to-Image Generation System
Diffusion models have opened up new avenues for the field of image
generation, resulting in the proliferation of high-quality models shared on
open-source platforms. However, a major challenge persists in current
text-to-image systems are often unable to handle diverse inputs, or are limited
to single model results. Current unified attempts often fall into two
orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
expert model to output. To combine the best of both worlds, we propose
DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
generation system capable of seamlessly accommodating various types of prompts
and integrating domain-expert models. DiffusionGPT constructs domain-specific
Trees for various generative models based on prior knowledge. When provided
with an input, the LLM parses the prompt and employs the Trees-of-Thought to
guide the selection of an appropriate model, thereby relaxing input constraints
and ensuring exceptional performance across diverse domains. Moreover, we
introduce Advantage Databases, where the Tree-of-Thought is enriched with human
feedback, aligning the model selection process with human preferences. Through
extensive experiments and comparisons, we demonstrate the effectiveness of
DiffusionGPT, showcasing its potential for pushing the boundaries of image
synthesis in diverse domains.
65a9e21dd766c6c6409f33eb
Jie Qin
65a9e21dd766c6c6409f33ec
Jie Wu
claimed_verified
2024-01-22T10:31:32.405Z
6381c5d63680a7cf34e08ca9
/avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
wujie10558@gmail.com
user
wujie10
65a9e21dd766c6c6409f33ed
Weifeng Chen
claimed_verified
2024-04-26T07:42:21.798Z
62a7fc03ac97233f161c2433
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7fc03ac97233f161c2433/19wL1aCwyR-ik2WDXvns9.jpeg
weifeng
user
weifeng-chen
65a9e21dd766c6c6409f33ee
Yuxi Ren
65a9e21dd766c6c6409f33ef
Huixia Li
claimed_verified
2024-02-07T11:19:19.974Z
6368b457e943416fc80619fb
https://cdn-avatars.huggingface.co/v1/production/uploads/6368b457e943416fc80619fb/RjAJKMwl6aIFKAZs1vgvK.jpeg
HXLee
user
HXLee
65a9e21dd766c6c6409f33f0
Hefeng Wu
65a9e21dd766c6c6409f33f1
Xuefeng Xiao
admin_assigned
2024-01-19T13:15:02.171Z
646b7f71df2609a541c1ab9f
/avatars/48b82e5fd9b06f41ff825507c36816cd.svg
Xuefeng Xiao
user
xiaoxuefeng
65a9e21dd766c6c6409f33f2
Rui Wang
65a9e21dd766c6c6409f33f3
Shilei Wen
admin_assigned
2024-01-19T13:14:53.634Z
6501763d12c1442d9964f883
/avatars/3e06bb6afdecc0ab8ac9a5cd215c0265.svg
Shileiwen
user
Shileiwen
DiffusionGPT: LLM-Driven Text-to-Image Generation System
Diffusion models have opened up new avenues for the field of image
generation, resulting in the proliferation of high-quality models shared on
open-source platforms. However, a major challenge persists in current
text-to-image systems are often unable to handle diverse inputs, or are limited
to single model results. Current unified attempts often fall into two
orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
expert model to output. To combine the best of both worlds, we propose
DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
generation system capable of seamlessly accommodating various types of prompts
and integrating domain-expert models. DiffusionGPT constructs domain-specific
Trees for various generative models based on prior knowledge. When provided
with an input, the LLM parses the prompt and employs the Trees-of-Thought to
guide the selection of an appropriate model, thereby relaxing input constraints
and ensuring exceptional performance across diverse domains. Moreover, we
introduce Advantage Databases, where the Tree-of-Thought is enriched with human
feedback, aligning the model selection process with human preferences. Through
extensive experiments and comparisons, we demonstrate the effectiveness of
DiffusionGPT, showcasing its potential for pushing the boundaries of image
synthesis in diverse domains.
DiffusionGPT combines Large Language Models with diffusion models and Trees-of-Thought to handle diverse prompts and integrate domain-expert models, demonstrating superior image synthesis performance across various domains.
diffusion models Large Language Models Trees-of-Thought domain-specific Trees Trees-of-Thought Advantage Databases