title: LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven
    Language Representation
summary: In this paper, we introduce LDGen, a novel method for integrating large
  language models (LLMs) into existing text-to-image diffusion models while
  minimizing computational demands. Traditional text encoders, such as CLIP and
  T5, exhibit limitations in multilingual processing, hindering image generation
  across diverse languages. We address these challenges by leveraging the
  advanced capabilities of LLMs. Our approach employs a language representation
  strategy that applies hierarchical caption optimization and human instruction
  techniques to derive precise semantic information,. Subsequently, we
  incorporate a lightweight adapter and a cross-modal refiner to facilitate
  efficient feature alignment and interaction between LLMs and image features.
  LDGen reduces training time and enables zero-shot multilingual image
  generation. Experimental results indicate that our method surpasses baseline
  models in both prompt adherence and image aesthetic quality, while seamlessly
  supporting multiple languages. Project page: https://zrealli.github.io/LDGen.
paper_authors._id: 67befd09afb202a5b7518572
paper_authors.name: Pengzhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T15:37:26.179Z
paper_authors._id: 64edcb9b84cc47a8b50bfab7
paper_authors.avatarUrl: /avatars/1b4defb79eef3753a540efa76c16462a.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: Kinpz
paper_authors._id: 67befd09afb202a5b7518573
paper_authors.name: Pengfei Yu
paper_authors._id: 67befd09afb202a5b7518574
paper_authors.name: Zide Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T13:08:16.002Z
paper_authors._id: 6431974f034ecbefddd4b463
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O6yDcmDTZk9IH2wIbuCQ3.jpeg
paper_authors.fullname: 刘自得
paper_authors.type: user
paper_authors.user: zideliu
paper_authors._id: 67befd09afb202a5b7518575
paper_authors.name: Wei He
paper_authors._id: 67befd09afb202a5b7518576
paper_authors.name: Xuhao Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T13:09:11.684Z
paper_authors._id: 64478f36a478b20f17543240
paper_authors.avatarUrl: /avatars/d5d40abc458282852cb758c6ca664038.svg
paper_authors.fullname: Pan Xuhao
paper_authors.type: user
paper_authors.user: qingtian0
paper_authors._id: 67befd09afb202a5b7518577
paper_authors.name: Xudong Rao
paper_authors._id: 67befd09afb202a5b7518578
paper_authors.name: Tao Wei
paper_authors._id: 67befd09afb202a5b7518579
paper_authors.name: Wei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T13:09:29.481Z
paper_authors._id: 632acc6b7fb39c2b6351d84e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632acc6b7fb39c2b6351d84e/2_VLtRuRUl-hNW4a89qUp.jpeg
paper_authors.fullname: wei chen
paper_authors.type: user
paper_authors.user: weichen
paper_title: LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven
    Language Representation
paper_summary: In this paper, we introduce LDGen, a novel method for integrating large
  language models (LLMs) into existing text-to-image diffusion models while
  minimizing computational demands. Traditional text encoders, such as CLIP and
  T5, exhibit limitations in multilingual processing, hindering image generation
  across diverse languages. We address these challenges by leveraging the
  advanced capabilities of LLMs. Our approach employs a language representation
  strategy that applies hierarchical caption optimization and human instruction
  techniques to derive precise semantic information,. Subsequently, we
  incorporate a lightweight adapter and a cross-modal refiner to facilitate
  efficient feature alignment and interaction between LLMs and image features.
  LDGen reduces training time and enables zero-shot multilingual image
  generation. Experimental results indicate that our method surpasses baseline
  models in both prompt adherence and image aesthetic quality, while seamlessly
  supporting multiple languages. Project page: https://zrealli.github.io/LDGen.
paper_ai_summary: LDGen integrates large language models into text-to-image diffusion models to achieve efficient zero-shot multilingual image generation with improved prompt adherence and aesthetic quality.
paper_ai_keywords: LLMs text-to-image diffusion models CLIP T5 multilingual processing language representation hierarchical caption optimization human instruction lightweight adapter cross-modal refiner feature alignment zero-shot multilingual image generation prompt adherence image aesthetic quality