Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with
  Large Vision-Language Model Support
Recent advancements in text-to-image models have significantly enhanced image
generation capabilities, yet a notable gap of open-source models persists in
bilingual or Chinese language support. To address this need, we present
Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
which is developed by extending the capabilities of CLIP and
Stable-Diffusion-XL through a process of bilingual continuous pre-training.
This approach includes the efficient expansion of vocabulary by integrating the
most frequently used Chinese characters into CLIP's tokenizer and embedding
layers, coupled with an absolute position encoding expansion. Additionally, we
enrich text prompts by large vision-language model, leading to better images
captions and possess higher visual quality. These enhancements are subsequently
applied to downstream text-to-image models. Our empirical results indicate that
the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
previous models. This research leads to the development and open-sourcing of
the Taiyi-Diffusion-XL model, representing a notable advancement in the field
of image generation, particularly for Chinese language applications. This
contribution is a step forward in addressing the need for more diverse language
support in multimodal research. The model and demonstration are made publicly
available at
https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/{this
https URL}, fostering further research and collaboration in this domain.
65b70e23e2510e886d93947b
Xiaojun Wu
claimed_verified
2024-01-29T11:43:44.936Z
6352637d0f9bdb641c44e52d
https://cdn-avatars.huggingface.co/v1/production/uploads/6352637d0f9bdb641c44e52d/mSBRPzcH5pIV68PUmcsHV.png
wuxiaojun
user
wuxiaojun
65b70e23e2510e886d93947c
Dixiang Zhang
65b70e23e2510e886d93947d
Ruyi Gan
admin_assigned
2024-01-29T11:18:24.585Z
612300897d70fa0063f38731
https://cdn-avatars.huggingface.co/v1/production/uploads/1637806723246-612300897d70fa0063f38731.jpeg
ganruyi
user
roygan
65b70e23e2510e886d93947e
Junyu Lu
65b70e23e2510e886d93947f
Ziwei Wu
admin_assigned
2024-01-29T11:18:56.672Z
619f0825678725e00cd9a27f
https://cdn-avatars.huggingface.co/v1/production/uploads/1637812566433-619f0825678725e00cd9a27f.png
wuziwei
user
Zimix
65b70e23e2510e886d939480
Renliang Sun
admin_assigned
2024-01-29T11:19:20.745Z
653a195b0da86d726c9c580c
/avatars/61649e1d600fdc1edc50ead0dfa99fdd.svg
Renliang Sun
user
RLSNLP
65b70e23e2510e886d939481
Jiaxing Zhang
admin_assigned
2024-01-29T11:19:27.767Z
637854b38f288aba3d01d261
/avatars/a7f0daf90326bdfea134c75b0bc9bea1.svg
Jiaxing Zhang
user
ideazhangjason
65b70e23e2510e886d939482
Pingjian Zhang
65b70e23e2510e886d939483
Yan Song
Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with
  Large Vision-Language Model Support
Recent advancements in text-to-image models have significantly enhanced image
generation capabilities, yet a notable gap of open-source models persists in
bilingual or Chinese language support. To address this need, we present
Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
which is developed by extending the capabilities of CLIP and
Stable-Diffusion-XL through a process of bilingual continuous pre-training.
This approach includes the efficient expansion of vocabulary by integrating the
most frequently used Chinese characters into CLIP's tokenizer and embedding
layers, coupled with an absolute position encoding expansion. Additionally, we
enrich text prompts by large vision-language model, leading to better images
captions and possess higher visual quality. These enhancements are subsequently
applied to downstream text-to-image models. Our empirical results indicate that
the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
previous models. This research leads to the development and open-sourcing of
the Taiyi-Diffusion-XL model, representing a notable advancement in the field
of image generation, particularly for Chinese language applications. This
contribution is a step forward in addressing the need for more diverse language
support in multimodal research. The model and demonstration are made publicly
available at
https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/{this
https URL}, fostering further research and collaboration in this domain.
Taiyi-Diffusion-XL is a bilingual text-to-image model that extends CLIP and Stable-Diffusion-XL for Chinese and English, improving vocabulary and vision-language capabilities, and achieving superior bilingual image generation.
text-to-image models CLIP Stable-Diffusion-XL bilingual continuous pre-training tokenizer embedding layers absolute position encoding large vision-language model image-text retrieval bilingual image generation