title: SketchVideo: Sketch-based Video Generation and Editing
summary: Video generation and editing conditioned on text prompts or images have
  undergone significant advancements. However, challenges remain in accurately
  controlling global layout and geometry details solely by texts, and supporting
  motion control and local modification through images. In this paper, we aim to
  achieve sketch-based spatial and motion control for video generation and
  support fine-grained editing of real or synthetic videos. Based on the DiT
  video generation model, we propose a memory-efficient control structure with
  sketch control blocks that predict residual features of skipped DiT blocks.
  Sketches are drawn on one or two keyframes (at arbitrary time points) for easy
  interaction. To propagate such temporally sparse sketch conditions across all
  frames, we propose an inter-frame attention mechanism to analyze the
  relationship between the keyframes and each video frame. For sketch-based video
  editing, we design an additional video insertion module that maintains
  consistency between the newly edited content and the original video's spatial
  feature and dynamic motion. During inference, we use latent fusion for the
  accurate preservation of unedited regions. Extensive experiments demonstrate
  that our SketchVideo achieves superior performance in controllable video
  generation and editing.
paper_authors._id: 67eb5280aeab4ce97de07134
paper_authors.name: Feng-Lin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:47:05.907Z
paper_authors._id: 6424538b9f9e65b42389920e
paper_authors.avatarUrl: /avatars/9b912e2af9eebe9a481181f006765059.svg
paper_authors.fullname: Feng-Lin Liu
paper_authors.type: user
paper_authors.user: Okrin
paper_authors._id: 67eb5280aeab4ce97de07135
paper_authors.name: Hongbo Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:07:01.616Z
paper_authors._id: 662cd8b9322afcbae53fb06e
paper_authors.avatarUrl: /avatars/9847f5c2282d49e61e76a0a303e0b2b1.svg
paper_authors.fullname: fuhongbo
paper_authors.type: user
paper_authors.user: fuhongbo
paper_authors._id: 67eb5280aeab4ce97de07136
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:07:09.910Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 67eb5280aeab4ce97de07137
paper_authors.name: Weicai Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:07:16.323Z
paper_authors._id: 6360d9f0472131c3bc4f61df
paper_authors.avatarUrl: /avatars/c5d884e5ef19b781e3405aba6dd68ca8.svg
paper_authors.fullname: WeicaiYe
paper_authors.type: user
paper_authors.user: WeicaiYe
paper_authors._id: 67eb5280aeab4ce97de07138
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:02.682Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67eb5280aeab4ce97de07139
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:07:30.236Z
paper_authors._id: 644c8324f02250233d0d67d9
paper_authors.avatarUrl: /avatars/feb39d281457c1750f3eada3c060a23e.svg
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: dizhang
paper_authors._id: 67eb5280aeab4ce97de0713a
paper_authors.name: Lin Gao
paper_title: SketchVideo: Sketch-based Video Generation and Editing
paper_summary: Video generation and editing conditioned on text prompts or images have
  undergone significant advancements. However, challenges remain in accurately
  controlling global layout and geometry details solely by texts, and supporting
  motion control and local modification through images. In this paper, we aim to
  achieve sketch-based spatial and motion control for video generation and
  support fine-grained editing of real or synthetic videos. Based on the DiT
  video generation model, we propose a memory-efficient control structure with
  sketch control blocks that predict residual features of skipped DiT blocks.
  Sketches are drawn on one or two keyframes (at arbitrary time points) for easy
  interaction. To propagate such temporally sparse sketch conditions across all
  frames, we propose an inter-frame attention mechanism to analyze the
  relationship between the keyframes and each video frame. For sketch-based video
  editing, we design an additional video insertion module that maintains
  consistency between the newly edited content and the original video's spatial
  feature and dynamic motion. During inference, we use latent fusion for the
  accurate preservation of unedited regions. Extensive experiments demonstrate
  that our SketchVideo achieves superior performance in controllable video
  generation and editing.
paper_ai_summary: SketchVideo enhances video generation and editing by incorporating sketch control blocks, inter-frame attention, and latent fusion, improving global layout, motion control, and consistency.
paper_ai_keywords: DiT memory-efficient control structure sketch control blocks keyframes inter-frame attention video insertion module latent fusion controllable video generation video editing