DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for
  Customized Manga Generation
Story visualization, the task of creating visual narratives from textual
descriptions, has seen progress with text-to-image generation models. However,
these models often lack effective control over character appearances and
interactions, particularly in multi-character scenes. To address these
limitations, we propose a new task: customized manga generation and
introduce DiffSensei, an innovative framework specifically designed
for generating manga with dynamic multi-character control. DiffSensei
integrates a diffusion-based image generator with a multimodal large language
model (MLLM) that acts as a text-compatible identity adapter. Our approach
employs masked cross-attention to seamlessly incorporate character features,
enabling precise layout control without direct pixel transfer. Additionally,
the MLLM-based adapter adjusts character features to align with panel-specific
text cues, allowing flexible adjustments in character expressions, poses, and
actions. We also introduce MangaZero, a large-scale dataset tailored
to this task, containing 43,264 manga pages and 427,147 annotated panels,
supporting the visualization of varied character interactions and movements
across sequential frames. Extensive experiments demonstrate that DiffSensei
outperforms existing models, marking a significant advancement in manga
generation by enabling text-adaptable character customization. The project page
is https://jianzongwu.github.io/projects/diffsensei/.
6759025d301d7a7b7ef45ee0
Jianzong Wu
admin_assigned
2024-12-11T10:12:40.931Z
657a6eed1ccc3c2a5ea7b585
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RIQIF-JJdNI0SwJEq_9z7.jpeg
Jianzong Wu
user
jianzongwu
6759025d301d7a7b7ef45ee1
Chao Tang
claimed_verified
2024-12-11T13:39:16.104Z
65bce64b8467e2a3d6a450af
https://cdn-avatars.huggingface.co/v1/production/uploads/65bce64b8467e2a3d6a450af/IFjoy4GYMA6oUgYyJfZ1F.jpeg
Chao Tang
user
Tangc03
6759025d301d7a7b7ef45ee2
Jingbo Wang
admin_assigned
2024-12-11T10:12:10.281Z
665590ac9ccb17d9670addce
/avatars/e13aa166af6c8ed9ee2fbb0add46b8dd.svg
Jingbo Wang
user
JingboWang
6759025d301d7a7b7ef45ee3
Yanhong Zeng
admin_assigned
2024-12-11T10:11:44.658Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
6759025d301d7a7b7ef45ee4
Xiangtai Li
admin_assigned
2024-12-11T10:11:39.250Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
6759025d301d7a7b7ef45ee5
Yunhai Tong
DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for
  Customized Manga Generation
Story visualization, the task of creating visual narratives from textual
descriptions, has seen progress with text-to-image generation models. However,
these models often lack effective control over character appearances and
interactions, particularly in multi-character scenes. To address these
limitations, we propose a new task: customized manga generation and
introduce DiffSensei, an innovative framework specifically designed
for generating manga with dynamic multi-character control. DiffSensei
integrates a diffusion-based image generator with a multimodal large language
model (MLLM) that acts as a text-compatible identity adapter. Our approach
employs masked cross-attention to seamlessly incorporate character features,
enabling precise layout control without direct pixel transfer. Additionally,
the MLLM-based adapter adjusts character features to align with panel-specific
text cues, allowing flexible adjustments in character expressions, poses, and
actions. We also introduce MangaZero, a large-scale dataset tailored
to this task, containing 43,264 manga pages and 427,147 annotated panels,
supporting the visualization of varied character interactions and movements
across sequential frames. Extensive experiments demonstrate that DiffSensei
outperforms existing models, marking a significant advancement in manga
generation by enabling text-adaptable character customization. The project page
is https://jianzongwu.github.io/projects/diffsensei/.
DiffSensei, a diffusion-based framework with a multimodal large language model, enhances manga generation by precisely controlling multi-character appearances and interactions using masked cross-attention and text cues.
diffusion-based multimodal large language model masked cross-attention manga generation text-adaptable character customization