title: MagicStick: Controllable Video Editing via Control Handle
    Transformations
summary: Text-based video editing has recently attracted considerable interest in
  changing the style or replacing the objects with a similar structure. Beyond
  this, we demonstrate that properties such as shape, size, location, motion,
  etc., can also be edited in videos. Our key insight is that the keyframe
  transformations of the specific internal feature (e.g., edge maps of objects or
  human pose), can easily propagate to other frames to provide generation
  guidance. We thus propose MagicStick, a controllable video editing method that
  edits the video properties by utilizing the transformation on the extracted
  internal control signals. In detail, to keep the appearance, we inflate both
  the pretrained image diffusion model and ControlNet to the temporal dimension
  and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in
  editing, we perform an inversion and editing framework. Differently, finetuned
  ControlNet is introduced in both inversion and generation for attention
  guidance with the proposed attention remix between the spatial attention maps
  of inversion and editing. Yet succinct, our method is the first method to show
  the ability of video property editing from the pre-trained text-to-image model.
  We present experiments on numerous examples within our unified framework. We
  also compare with shape-aware text-based editing and handcrafted motion video
  generation, demonstrating our superior temporal consistency and editing
  capability than previous works. The code and models will be made publicly
  available.
paper_authors._id: 65713489c8018fe6408d9ba4
paper_authors.name: Yue Ma
paper_authors._id: 65713489c8018fe6408d9ba5
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:00:42.458Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 65713489c8018fe6408d9ba6
paper_authors.name: Yingqing He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:00:52.831Z
paper_authors._id: 630231bf7e137e3d6b3b0645
paper_authors.avatarUrl: /avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
paper_authors.fullname: He
paper_authors.type: user
paper_authors.user: Yingqing
paper_authors._id: 65713489c8018fe6408d9ba7
paper_authors.name: Chenyang Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:01:03.648Z
paper_authors._id: 642157a201c62c1e41fc526a
paper_authors.avatarUrl: /avatars/73ae831a73947a04468b1e87d17da24d.svg
paper_authors.fullname: chenyangqi
paper_authors.type: user
paper_authors.user: chenyangqi
paper_authors._id: 65713489c8018fe6408d9ba8
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:01:13.305Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 65713489c8018fe6408d9ba9
paper_authors.name: Ying Shan
paper_authors._id: 65713489c8018fe6408d9baa
paper_authors.name: Xiu Li
paper_authors._id: 65713489c8018fe6408d9bab
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:01:44.912Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_title: MagicStick: Controllable Video Editing via Control Handle
    Transformations
paper_summary: Text-based video editing has recently attracted considerable interest in
  changing the style or replacing the objects with a similar structure. Beyond
  this, we demonstrate that properties such as shape, size, location, motion,
  etc., can also be edited in videos. Our key insight is that the keyframe
  transformations of the specific internal feature (e.g., edge maps of objects or
  human pose), can easily propagate to other frames to provide generation
  guidance. We thus propose MagicStick, a controllable video editing method that
  edits the video properties by utilizing the transformation on the extracted
  internal control signals. In detail, to keep the appearance, we inflate both
  the pretrained image diffusion model and ControlNet to the temporal dimension
  and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in
  editing, we perform an inversion and editing framework. Differently, finetuned
  ControlNet is introduced in both inversion and generation for attention
  guidance with the proposed attention remix between the spatial attention maps
  of inversion and editing. Yet succinct, our method is the first method to show
  the ability of video property editing from the pre-trained text-to-image model.
  We present experiments on numerous examples within our unified framework. We
  also compare with shape-aware text-based editing and handcrafted motion video
  generation, demonstrating our superior temporal consistency and editing
  capability than previous works. The code and models will be made publicly
  available.
paper_ai_keywords: pretrained image diffusion model ControlNet low-rank adaptions (LORA) spatial attention maps temporal dimension Neural editing attention remix temporal consistency text-to-image model