Visual Text Generation in the Wild
Recently, with the rapid advancements of generative models, the field of
visual text generation has witnessed significant progress. However, it is still
challenging to render high-quality text images in real-world scenarios, as
three critical criteria should be satisfied: (1) Fidelity: the generated text
images should be photo-realistic and the contents are expected to be the same
as specified in the given conditions; (2) Reasonability: the regions and
contents of the generated text should cohere with the scene; (3) Utility: the
generated text images can facilitate related tasks (e.g., text detection and
recognition). Upon investigation, we find that existing methods, either
rendering-based or diffusion-based, can hardly meet all these aspects
simultaneously, limiting their application range. Therefore, we propose in this
paper a visual text generator (termed SceneVTG), which can produce high-quality
text images in the wild. Following a two-stage paradigm, SceneVTG leverages a
Multimodal Large Language Model to recommend reasonable text regions and
contents across multiple scales and levels, which are used by a conditional
diffusion model as conditions to generate text images. Extensive experiments
demonstrate that the proposed SceneVTG significantly outperforms traditional
rendering-based methods and recent diffusion-based methods in terms of fidelity
and reasonability. Besides, the generated images provide superior utility for
tasks involving text detection and text recognition. Code and datasets are
available at AdvancedLiterateMachinery.
669dc4af811696bba807562c
Yuanzhi Zhu
admin_assigned
2024-07-22T10:06:34.349Z
627d2723401f42c57b6b7c0c
/avatars/6ff754e56aaee63d8572881a6a966171.svg
Yuanzhi Zhu
user
Yuanzhi
669dc4af811696bba807562d
Jiawei Liu
admin_assigned
2024-07-22T10:06:52.161Z
64230048a73327caad9d8241
/avatars/ac115f1a21c743db7c925f0e18451145.svg
Jiawei Liu
user
JiaweiLIU
669dc4af811696bba807562e
Feiyu Gao
669dc4af811696bba807562f
Wenyu Liu
669dc4af811696bba8075630
Xinggang Wang
admin_assigned
2024-07-22T10:10:14.794Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
669dc4af811696bba8075631
Peng Wang
669dc4af811696bba8075632
Fei Huang
admin_assigned
2024-07-22T10:11:58.528Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
669dc4af811696bba8075633
Cong Yao
admin_assigned
2024-07-22T10:12:05.099Z
6638f898c168eec02354abeb
/avatars/0a0b7a1b056463fb4ccf6bfe6456a7d1.svg
Cong Yao
user
bridgetop3young
669dc4af811696bba8075634
Zhibo Yang
admin_assigned
2024-07-22T10:12:13.078Z
66051de91b8220d71becb448
/avatars/f2b507f5821f8baa90b60f62ae4de389.svg
杨智博
user
zhiboyang
Visual Text Generation in the Wild
Recently, with the rapid advancements of generative models, the field of
visual text generation has witnessed significant progress. However, it is still
challenging to render high-quality text images in real-world scenarios, as
three critical criteria should be satisfied: (1) Fidelity: the generated text
images should be photo-realistic and the contents are expected to be the same
as specified in the given conditions; (2) Reasonability: the regions and
contents of the generated text should cohere with the scene; (3) Utility: the
generated text images can facilitate related tasks (e.g., text detection and
recognition). Upon investigation, we find that existing methods, either
rendering-based or diffusion-based, can hardly meet all these aspects
simultaneously, limiting their application range. Therefore, we propose in this
paper a visual text generator (termed SceneVTG), which can produce high-quality
text images in the wild. Following a two-stage paradigm, SceneVTG leverages a
Multimodal Large Language Model to recommend reasonable text regions and
contents across multiple scales and levels, which are used by a conditional
diffusion model as conditions to generate text images. Extensive experiments
demonstrate that the proposed SceneVTG significantly outperforms traditional
rendering-based methods and recent diffusion-based methods in terms of fidelity
and reasonability. Besides, the generated images provide superior utility for
tasks involving text detection and text recognition. Code and datasets are
available at AdvancedLiterateMachinery.
SceneVTG combines a Multimodal Large Language Model and a conditional diffusion model to generate high-quality, photo-realistic, and contextually coherent text images, surpassing existing methods in fidelity, reasonability, and utility for text detection and recognition.
Multimodal Large Language Model conditional diffusion model