title: LVCD: Reference-based Lineart Video Colorization with Diffusion Models
summary: We propose the first video diffusion framework for reference-based lineart
  video colorization. Unlike previous works that rely solely on image generative
  models to colorize lineart frame by frame, our approach leverages a large-scale
  pretrained video diffusion model to generate colorized animation videos. This
  approach leads to more temporally consistent results and is better equipped to
  handle large motions. Firstly, we introduce Sketch-guided ControlNet which
  provides additional control to finetune an image-to-video diffusion model for
  controllable video synthesis, enabling the generation of animation videos
  conditioned on lineart. We then propose Reference Attention to facilitate the
  transfer of colors from the reference frame to other frames containing fast and
  expansive motions. Finally, we present a novel scheme for sequential sampling,
  incorporating the Overlapped Blending Module and Prev-Reference Attention, to
  extend the video diffusion model beyond its original fixed-length limitation
  for long video colorization. Both qualitative and quantitative results
  demonstrate that our method significantly outperforms state-of-the-art
  techniques in terms of frame and video quality, as well as temporal
  consistency. Moreover, our method is capable of generating high-quality, long
  temporal-consistent animation videos with large motions, which is not
  achievable in previous works. Our code and model are available at
  https://luckyhzt.github.io/lvcd.
paper_authors._id: 66ecded3381b190036da5ad8
paper_authors.name: Zhitong Huang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-20T02:38:12.123Z
paper_authors._id: 6321afb13007fcbf292ac1f4
paper_authors.avatarUrl: /avatars/fb417553249580f1fb35bc127f550a5f.svg
paper_authors.fullname: Zhitong Huang
paper_authors.type: user
paper_authors.user: luckyhzt
paper_authors._id: 66ecded3381b190036da5ad9
paper_authors.name: Mohan Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-26T01:24:55.221Z
paper_authors._id: 66f227dd0c26057904ede944
paper_authors.avatarUrl: /avatars/49ee01a747afd00ba8b337980d92cc64.svg
paper_authors.fullname: mohan zhang
paper_authors.type: user
paper_authors.user: zeromhzhang
paper_authors._id: 66ecded3381b190036da5ada
paper_authors.name: Jing Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:28:46.446Z
paper_authors._id: 65e77726767bfc7d109c45bf
paper_authors.avatarUrl: /avatars/24e68c86e06055ea1209598ba49ce8b9.svg
paper_authors.fullname: Jing Liao
paper_authors.type: user
paper_authors.user: CeciliaJL
paper_title: LVCD: Reference-based Lineart Video Colorization with Diffusion Models
paper_summary: We propose the first video diffusion framework for reference-based lineart
  video colorization. Unlike previous works that rely solely on image generative
  models to colorize lineart frame by frame, our approach leverages a large-scale
  pretrained video diffusion model to generate colorized animation videos. This
  approach leads to more temporally consistent results and is better equipped to
  handle large motions. Firstly, we introduce Sketch-guided ControlNet which
  provides additional control to finetune an image-to-video diffusion model for
  controllable video synthesis, enabling the generation of animation videos
  conditioned on lineart. We then propose Reference Attention to facilitate the
  transfer of colors from the reference frame to other frames containing fast and
  expansive motions. Finally, we present a novel scheme for sequential sampling,
  incorporating the Overlapped Blending Module and Prev-Reference Attention, to
  extend the video diffusion model beyond its original fixed-length limitation
  for long video colorization. Both qualitative and quantitative results
  demonstrate that our method significantly outperforms state-of-the-art
  techniques in terms of frame and video quality, as well as temporal
  consistency. Moreover, our method is capable of generating high-quality, long
  temporal-consistent animation videos with large motions, which is not
  achievable in previous works. Our code and model are available at
  https://luckyhzt.github.io/lvcd.
paper_ai_keywords: video diffusion framework reference-based lineart video colorization video diffusion model Sketch-guided ControlNet image-to-video diffusion model controllable video synthesis Reference Attention sequential sampling Overlapped Blending Module Prev-Reference Attention fixed-length limitation long video colorization temporal consistency frame and video quality