title: Visual Text Generation in the Wild
summary: Recently, with the rapid advancements of generative models, the field of
  visual text generation has witnessed significant progress. However, it is still
  challenging to render high-quality text images in real-world scenarios, as
  three critical criteria should be satisfied: (1) Fidelity: the generated text
  images should be photo-realistic and the contents are expected to be the same
  as specified in the given conditions; (2) Reasonability: the regions and
  contents of the generated text should cohere with the scene; (3) Utility: the
  generated text images can facilitate related tasks (e.g., text detection and
  recognition). Upon investigation, we find that existing methods, either
  rendering-based or diffusion-based, can hardly meet all these aspects
  simultaneously, limiting their application range. Therefore, we propose in this
  paper a visual text generator (termed SceneVTG), which can produce high-quality
  text images in the wild. Following a two-stage paradigm, SceneVTG leverages a
  Multimodal Large Language Model to recommend reasonable text regions and
  contents across multiple scales and levels, which are used by a conditional
  diffusion model as conditions to generate text images. Extensive experiments
  demonstrate that the proposed SceneVTG significantly outperforms traditional
  rendering-based methods and recent diffusion-based methods in terms of fidelity
  and reasonability. Besides, the generated images provide superior utility for
  tasks involving text detection and text recognition. Code and datasets are
  available at AdvancedLiterateMachinery.
paper_authors._id: 669dc4af811696bba807562c
paper_authors.name: Yuanzhi Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:06:34.349Z
paper_authors._id: 627d2723401f42c57b6b7c0c
paper_authors.avatarUrl: /avatars/6ff754e56aaee63d8572881a6a966171.svg
paper_authors.fullname: Yuanzhi Zhu
paper_authors.type: user
paper_authors.user: Yuanzhi
paper_authors._id: 669dc4af811696bba807562d
paper_authors.name: Jiawei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:06:52.161Z
paper_authors._id: 64230048a73327caad9d8241
paper_authors.avatarUrl: /avatars/ac115f1a21c743db7c925f0e18451145.svg
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: JiaweiLIU
paper_authors._id: 669dc4af811696bba807562e
paper_authors.name: Feiyu Gao
paper_authors._id: 669dc4af811696bba807562f
paper_authors.name: Wenyu Liu
paper_authors._id: 669dc4af811696bba8075630
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:10:14.794Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_authors._id: 669dc4af811696bba8075631
paper_authors.name: Peng Wang
paper_authors._id: 669dc4af811696bba8075632
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:11:58.528Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 669dc4af811696bba8075633
paper_authors.name: Cong Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:12:05.099Z
paper_authors._id: 6638f898c168eec02354abeb
paper_authors.avatarUrl: /avatars/0a0b7a1b056463fb4ccf6bfe6456a7d1.svg
paper_authors.fullname: Cong Yao
paper_authors.type: user
paper_authors.user: bridgetop3young
paper_authors._id: 669dc4af811696bba8075634
paper_authors.name: Zhibo Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:12:13.078Z
paper_authors._id: 66051de91b8220d71becb448
paper_authors.avatarUrl: /avatars/f2b507f5821f8baa90b60f62ae4de389.svg
paper_authors.fullname: 杨智博
paper_authors.type: user
paper_authors.user: zhiboyang
paper_title: Visual Text Generation in the Wild
paper_summary: Recently, with the rapid advancements of generative models, the field of
  visual text generation has witnessed significant progress. However, it is still
  challenging to render high-quality text images in real-world scenarios, as
  three critical criteria should be satisfied: (1) Fidelity: the generated text
  images should be photo-realistic and the contents are expected to be the same
  as specified in the given conditions; (2) Reasonability: the regions and
  contents of the generated text should cohere with the scene; (3) Utility: the
  generated text images can facilitate related tasks (e.g., text detection and
  recognition). Upon investigation, we find that existing methods, either
  rendering-based or diffusion-based, can hardly meet all these aspects
  simultaneously, limiting their application range. Therefore, we propose in this
  paper a visual text generator (termed SceneVTG), which can produce high-quality
  text images in the wild. Following a two-stage paradigm, SceneVTG leverages a
  Multimodal Large Language Model to recommend reasonable text regions and
  contents across multiple scales and levels, which are used by a conditional
  diffusion model as conditions to generate text images. Extensive experiments
  demonstrate that the proposed SceneVTG significantly outperforms traditional
  rendering-based methods and recent diffusion-based methods in terms of fidelity
  and reasonability. Besides, the generated images provide superior utility for
  tasks involving text detection and text recognition. Code and datasets are
  available at AdvancedLiterateMachinery.
paper_ai_keywords: Multimodal Large Language Model conditional Diffusion model two-stage paradigm rendering-based methods diffusion-based methods text detection text recognition fidelity reasonability utility