MTV-Inpaint: Multi-Task Long Video Inpainting
Video inpainting involves modifying local regions within a video, ensuring
spatial and temporal consistency. Most existing methods focus primarily on
scene completion (i.e., filling missing regions) and lack the capability to
insert new objects into a scene in a controllable manner. Fortunately, recent
advancements in text-to-video (T2V) diffusion models pave the way for
text-guided video inpainting. However, directly adapting T2V models for
inpainting remains limited in unifying completion and insertion tasks, lacks
input controllability, and struggles with long videos, thereby restricting
their applicability and flexibility. To address these challenges, we propose
MTV-Inpaint, a unified multi-task video inpainting framework capable of
handling both traditional scene completion and novel object insertion tasks. To
unify these distinct tasks, we design a dual-branch spatial attention mechanism
in the T2V diffusion U-Net, enabling seamless integration of scene completion
and object insertion within a single framework. In addition to textual
guidance, MTV-Inpaint supports multimodal control by integrating various image
inpainting models through our proposed image-to-video (I2V) inpainting mode.
Additionally, we propose a two-stage pipeline that combines keyframe inpainting
with in-between frame propagation, enabling MTV-Inpaint to effectively handle
long videos with hundreds of frames. Extensive experiments demonstrate that
MTV-Inpaint achieves state-of-the-art performance in both scene completion and
object insertion tasks. Furthermore, it demonstrates versatility in derived
applications such as multi-modal inpainting, object editing, removal, image
object brush, and the ability to handle long videos. Project page:
https://mtv-inpaint.github.io/.
67d8f8b77f61dda9ea6512b7
Shiyuan Yang
67d8f8b77f61dda9ea6512b8
Zheng Gu
admin_assigned
2025-03-18T09:33:49.096Z
678f49878af7a399877b87c0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3IHnM4AKbIW_wmL15wdZf.png
GuZheng
user
GuZheng
67d8f8b77f61dda9ea6512b9
Liang Hou
admin_assigned
2025-03-18T09:33:57.501Z
64560a2aaaaf85a98fa9a4b9
https://cdn-avatars.huggingface.co/v1/production/uploads/64560a2aaaaf85a98fa9a4b9/2Kp0S0sMVpKqo81s-l_Yt.png
Liang Hou
user
lianghou
67d8f8b77f61dda9ea6512ba
Xin Tao
67d8f8b77f61dda9ea6512bb
Pengfei Wan
extracted_confirmed
2025-03-18T04:42:06.437Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67d8f8b77f61dda9ea6512bc
Xiaodong Chen
admin_assigned
2025-03-18T09:34:15.858Z
67be7cd21616162fc336cb44
/avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
Xiaodong Chen
user
XiaodongChen
67d8f8b77f61dda9ea6512bd
Jing Liao
admin_assigned
2025-03-18T09:34:23.653Z
65e77726767bfc7d109c45bf
/avatars/24e68c86e06055ea1209598ba49ce8b9.svg
Jing Liao
user
CeciliaJL
MTV-Inpaint: Multi-Task Long Video Inpainting
Video inpainting involves modifying local regions within a video, ensuring
spatial and temporal consistency. Most existing methods focus primarily on
scene completion (i.e., filling missing regions) and lack the capability to
insert new objects into a scene in a controllable manner. Fortunately, recent
advancements in text-to-video (T2V) diffusion models pave the way for
text-guided video inpainting. However, directly adapting T2V models for
inpainting remains limited in unifying completion and insertion tasks, lacks
input controllability, and struggles with long videos, thereby restricting
their applicability and flexibility. To address these challenges, we propose
MTV-Inpaint, a unified multi-task video inpainting framework capable of
handling both traditional scene completion and novel object insertion tasks. To
unify these distinct tasks, we design a dual-branch spatial attention mechanism
in the T2V diffusion U-Net, enabling seamless integration of scene completion
and object insertion within a single framework. In addition to textual
guidance, MTV-Inpaint supports multimodal control by integrating various image
inpainting models through our proposed image-to-video (I2V) inpainting mode.
Additionally, we propose a two-stage pipeline that combines keyframe inpainting
with in-between frame propagation, enabling MTV-Inpaint to effectively handle
long videos with hundreds of frames. Extensive experiments demonstrate that
MTV-Inpaint achieves state-of-the-art performance in both scene completion and
object insertion tasks. Furthermore, it demonstrates versatility in derived
applications such as multi-modal inpainting, object editing, removal, image
object brush, and the ability to handle long videos. Project page:
https://mtv-inpaint.github.io/.
MTV-Inpaint is a unified framework using T2V diffusion and spatial attention for both scene completion and object insertion in videos, with support for multimodal control and long video handling.
text-to-video (T2V) diffusion models dual-branch spatial attention mechanism T2V diffusion U-Net image-to-video (I2V) inpainting mode keyframe inpainting in-between frame propagation scene completion object insertion multimodal inpainting object editing object removal image object brush