title: Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with
    Large Vision-Language Model Support
summary: Recent advancements in text-to-image models have significantly enhanced image
  generation capabilities, yet a notable gap of open-source models persists in
  bilingual or Chinese language support. To address this need, we present
  Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
  which is developed by extending the capabilities of CLIP and
  Stable-Diffusion-XL through a process of bilingual continuous pre-training.
  This approach includes the efficient expansion of vocabulary by integrating the
  most frequently used Chinese characters into CLIP's tokenizer and embedding
  layers, coupled with an absolute position encoding expansion. Additionally, we
  enrich text prompts by large vision-language model, leading to better images
  captions and possess higher visual quality. These enhancements are subsequently
  applied to downstream text-to-image models. Our empirical results indicate that
  the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
  the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
  previous models. This research leads to the development and open-sourcing of
  the Taiyi-Diffusion-XL model, representing a notable advancement in the field
  of image generation, particularly for Chinese language applications. This
  contribution is a step forward in addressing the need for more diverse language
  support in multimodal research. The model and demonstration are made publicly
  available at
  https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/{this
  https URL}, fostering further research and collaboration in this domain.
paper_authors._id: 65b70e23e2510e886d93947b
paper_authors.name: Xiaojun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T11:43:44.936Z
paper_authors._id: 6352637d0f9bdb641c44e52d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6352637d0f9bdb641c44e52d/mSBRPzcH5pIV68PUmcsHV.png
paper_authors.fullname: wuxiaojun
paper_authors.type: user
paper_authors.user: wuxiaojun
paper_authors._id: 65b70e23e2510e886d93947c
paper_authors.name: Dixiang Zhang
paper_authors._id: 65b70e23e2510e886d93947d
paper_authors.name: Ruyi Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T11:18:24.585Z
paper_authors._id: 612300897d70fa0063f38731
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637806723246-612300897d70fa0063f38731.jpeg
paper_authors.fullname: ganruyi
paper_authors.type: user
paper_authors.user: roygan
paper_authors._id: 65b70e23e2510e886d93947e
paper_authors.name: Junyu Lu
paper_authors._id: 65b70e23e2510e886d93947f
paper_authors.name: Ziwei Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T11:18:56.672Z
paper_authors._id: 619f0825678725e00cd9a27f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637812566433-619f0825678725e00cd9a27f.png
paper_authors.fullname: wuziwei
paper_authors.type: user
paper_authors.user: Zimix
paper_authors._id: 65b70e23e2510e886d939480
paper_authors.name: Renliang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T11:19:20.745Z
paper_authors._id: 653a195b0da86d726c9c580c
paper_authors.avatarUrl: /avatars/61649e1d600fdc1edc50ead0dfa99fdd.svg
paper_authors.fullname: Renliang Sun
paper_authors.type: user
paper_authors.user: RLSNLP
paper_authors._id: 65b70e23e2510e886d939481
paper_authors.name: Jiaxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T11:19:27.767Z
paper_authors._id: 637854b38f288aba3d01d261
paper_authors.avatarUrl: /avatars/a7f0daf90326bdfea134c75b0bc9bea1.svg
paper_authors.fullname: Jiaxing Zhang
paper_authors.type: user
paper_authors.user: ideazhangjason
paper_authors._id: 65b70e23e2510e886d939482
paper_authors.name: Pingjian Zhang
paper_authors._id: 65b70e23e2510e886d939483
paper_authors.name: Yan Song
paper_title: Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with
    Large Vision-Language Model Support
paper_summary: Recent advancements in text-to-image models have significantly enhanced image
  generation capabilities, yet a notable gap of open-source models persists in
  bilingual or Chinese language support. To address this need, we present
  Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
  which is developed by extending the capabilities of CLIP and
  Stable-Diffusion-XL through a process of bilingual continuous pre-training.
  This approach includes the efficient expansion of vocabulary by integrating the
  most frequently used Chinese characters into CLIP's tokenizer and embedding
  layers, coupled with an absolute position encoding expansion. Additionally, we
  enrich text prompts by large vision-language model, leading to better images
  captions and possess higher visual quality. These enhancements are subsequently
  applied to downstream text-to-image models. Our empirical results indicate that
  the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
  the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
  previous models. This research leads to the development and open-sourcing of
  the Taiyi-Diffusion-XL model, representing a notable advancement in the field
  of image generation, particularly for Chinese language applications. This
  contribution is a step forward in addressing the need for more diverse language
  support in multimodal research. The model and demonstration are made publicly
  available at
  https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/{this
  https URL}, fostering further research and collaboration in this domain.
paper_ai_summary: Taiyi-Diffusion-XL is a bilingual text-to-image model that extends CLIP and Stable-Diffusion-XL for Chinese and English, improving vocabulary and vision-language capabilities, and achieving superior bilingual image generation.
paper_ai_keywords: text-to-image models CLIP Stable-Diffusion-XL bilingual continuous pre-training tokenizer embedding layers absolute position encoding large vision-language model image-text retrieval bilingual image generation