title: SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with
    Large Language Models
summary: Diffusion models, which have emerged to become popular text-to-image
  generation models, can produce high-quality and content-rich images guided by
  textual prompts. However, there are limitations to semantic understanding and
  commonsense reasoning in existing models when the input prompts are concise
  narrative, resulting in low-quality image generation. To improve the capacities
  for narrative prompts, we propose a simple-yet-effective parameter-efficient
  fine-tuning approach called the Semantic Understanding and Reasoning adapter
  (SUR-adapter) for pre-trained diffusion models. To reach this goal, we first
  collect and annotate a new dataset SURD which consists of more than 57,000
  semantically corrected multi-modal samples. Each sample contains a simple
  narrative prompt, a complex keyword-based prompt, and a high-quality image.
  Then, we align the semantic representation of narrative prompts to the complex
  prompts and transfer knowledge of large language models (LLMs) to our
  SUR-adapter via knowledge distillation so that it can acquire the powerful
  semantic understanding and reasoning capabilities to build a high-quality
  textual semantic representation for text-to-image generation. We conduct
  experiments by integrating multiple LLMs and popular pre-trained diffusion
  models to show the effectiveness of our approach in enabling diffusion models
  to understand and reason concise natural language without image quality
  degradation. Our approach can make text-to-image diffusion models easier to use
  with better user experience, which demonstrates our approach has the potential
  for further advancing the development of user-friendly text-to-image generation
  models by bridging the semantic gap between simple narrative prompts and
  complex keyword-based prompts.
paper_authors._id: 645aebd221ab438e732bf6f9
paper_authors.name: Shanshan Zhong
paper_authors._id: 645aebd221ab438e732bf6fa
paper_authors.name: Zhongzhan Huang
paper_authors._id: 645aebd221ab438e732bf6fb
paper_authors.name: Wushao Wen
paper_authors._id: 645aebd221ab438e732bf6fc
paper_authors.name: Jinghui Qin
paper_authors._id: 645aebd221ab438e732bf6fd
paper_authors.name: Liang Lin
paper_title: SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with
    Large Language Models
paper_summary: Diffusion models, which have emerged to become popular text-to-image
  generation models, can produce high-quality and content-rich images guided by
  textual prompts. However, there are limitations to semantic understanding and
  commonsense reasoning in existing models when the input prompts are concise
  narrative, resulting in low-quality image generation. To improve the capacities
  for narrative prompts, we propose a simple-yet-effective parameter-efficient
  fine-tuning approach called the Semantic Understanding and Reasoning adapter
  (SUR-adapter) for pre-trained diffusion models. To reach this goal, we first
  collect and annotate a new dataset SURD which consists of more than 57,000
  semantically corrected multi-modal samples. Each sample contains a simple
  narrative prompt, a complex keyword-based prompt, and a high-quality image.
  Then, we align the semantic representation of narrative prompts to the complex
  prompts and transfer knowledge of large language models (LLMs) to our
  SUR-adapter via knowledge distillation so that it can acquire the powerful
  semantic understanding and reasoning capabilities to build a high-quality
  textual semantic representation for text-to-image generation. We conduct
  experiments by integrating multiple LLMs and popular pre-trained diffusion
  models to show the effectiveness of our approach in enabling diffusion models
  to understand and reason concise natural language without image quality
  degradation. Our approach can make text-to-image diffusion models easier to use
  with better user experience, which demonstrates our approach has the potential
  for further advancing the development of user-friendly text-to-image generation
  models by bridging the semantic gap between simple narrative prompts and
  complex keyword-based prompts.
paper_ai_summary: A parameter-efficient fine-tuning method, SUR-adapter, is proposed to enhance the semantic understanding and reasoning of diffusion models for text-to-image generation, using knowledge distillation from large language models.
paper_ai_keywords: diffusion models parameter-efficient fine-tuning SUR-adapter semantic understanding reasoning narrative prompts complex prompts knowledge distillation large language models high-quality textual semantic representation