title: DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for
    Customized Manga Generation
summary: Story visualization, the task of creating visual narratives from textual
  descriptions, has seen progress with text-to-image generation models. However,
  these models often lack effective control over character appearances and
  interactions, particularly in multi-character scenes. To address these
  limitations, we propose a new task: customized manga generation and
  introduce DiffSensei, an innovative framework specifically designed
  for generating manga with dynamic multi-character control. DiffSensei
  integrates a diffusion-based image generator with a multimodal large language
  model (MLLM) that acts as a text-compatible identity adapter. Our approach
  employs masked cross-attention to seamlessly incorporate character features,
  enabling precise layout control without direct pixel transfer. Additionally,
  the MLLM-based adapter adjusts character features to align with panel-specific
  text cues, allowing flexible adjustments in character expressions, poses, and
  actions. We also introduce MangaZero, a large-scale dataset tailored
  to this task, containing 43,264 manga pages and 427,147 annotated panels,
  supporting the visualization of varied character interactions and movements
  across sequential frames. Extensive experiments demonstrate that DiffSensei
  outperforms existing models, marking a significant advancement in manga
  generation by enabling text-adaptable character customization. The project page
  is https://jianzongwu.github.io/projects/diffsensei/.
paper_authors._id: 6759025d301d7a7b7ef45ee0
paper_authors.name: Jianzong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:12:40.931Z
paper_authors._id: 657a6eed1ccc3c2a5ea7b585
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RIQIF-JJdNI0SwJEq_9z7.jpeg
paper_authors.fullname: Jianzong Wu
paper_authors.type: user
paper_authors.user: jianzongwu
paper_authors._id: 6759025d301d7a7b7ef45ee1
paper_authors.name: Chao Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T13:39:16.104Z
paper_authors._id: 65bce64b8467e2a3d6a450af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65bce64b8467e2a3d6a450af/ibrJanTW44ZzD_k-HnUEh.png
paper_authors.fullname: Chao Tang
paper_authors.type: user
paper_authors.user: Tangc03
paper_authors._id: 6759025d301d7a7b7ef45ee2
paper_authors.name: Jingbo Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:12:10.281Z
paper_authors._id: 665590ac9ccb17d9670addce
paper_authors.avatarUrl: /avatars/e13aa166af6c8ed9ee2fbb0add46b8dd.svg
paper_authors.fullname: Jingbo Wang
paper_authors.type: user
paper_authors.user: JingboWang
paper_authors._id: 6759025d301d7a7b7ef45ee3
paper_authors.name: Yanhong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:11:44.658Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 6759025d301d7a7b7ef45ee4
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:11:39.250Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 6759025d301d7a7b7ef45ee5
paper_authors.name: Yunhai Tong
paper_title: DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for
    Customized Manga Generation
paper_summary: Story visualization, the task of creating visual narratives from textual
  descriptions, has seen progress with text-to-image generation models. However,
  these models often lack effective control over character appearances and
  interactions, particularly in multi-character scenes. To address these
  limitations, we propose a new task: customized manga generation and
  introduce DiffSensei, an innovative framework specifically designed
  for generating manga with dynamic multi-character control. DiffSensei
  integrates a diffusion-based image generator with a multimodal large language
  model (MLLM) that acts as a text-compatible identity adapter. Our approach
  employs masked cross-attention to seamlessly incorporate character features,
  enabling precise layout control without direct pixel transfer. Additionally,
  the MLLM-based adapter adjusts character features to align with panel-specific
  text cues, allowing flexible adjustments in character expressions, poses, and
  actions. We also introduce MangaZero, a large-scale dataset tailored
  to this task, containing 43,264 manga pages and 427,147 annotated panels,
  supporting the visualization of varied character interactions and movements
  across sequential frames. Extensive experiments demonstrate that DiffSensei
  outperforms existing models, marking a significant advancement in manga
  generation by enabling text-adaptable character customization. The project page
  is https://jianzongwu.github.io/projects/diffsensei/.
paper_ai_keywords: diffusion-based image generator multimodal large language model (MLLM) masked cross-attention text-compatible identity adapter panel-specific text cues MangaZero character customization