MIVE: New Design and Benchmark for Multi-Instance Video Editing
Recent AI-based video editing has enabled users to edit videos through simple
text prompts, significantly simplifying the editing process. However, recent
zero-shot video editing techniques primarily focus on global or single-object
edits, which can lead to unintended changes in other parts of the video. When
multiple objects require localized edits, existing methods face challenges,
such as unfaithful editing, editing leakage, and lack of suitable evaluation
datasets and metrics. To overcome these limitations, we propose a zero-shot
Multi-Instance Video Editing
framework, called MIVE. MIVE is a general-purpose mask-based framework, not
dedicated to specific objects (e.g., people). MIVE introduces two key modules:
(i) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage and
(ii) Instance-centric Probability Redistribution (IPR) to ensure precise
localization and faithful editing. Additionally, we present our new MIVE
Dataset featuring diverse video scenarios and introduce the Cross-Instance
Accuracy (CIA) Score to evaluate editing leakage in multi-instance video
editing tasks. Our extensive qualitative, quantitative, and user study
evaluations demonstrate that MIVE significantly outperforms recent
state-of-the-art methods in terms of editing faithfulness, accuracy, and
leakage prevention, setting a new benchmark for multi-instance video editing.
The project page is available at https://kaist-viclab.github.io/mive-site/
67628a706562d8e0d34a6ed6
Samuel Teodoro
admin_assigned
2024-12-18T12:39:54.778Z
669ffd8bc8cb5058061229cf
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SIGrFQfuTTh1dMSh5um1P.png
Samuel Teodoro
user
TheRota
67628a706562d8e0d34a6ed7
Agus Gunawan
admin_assigned
2024-12-18T12:39:38.581Z
63bbeda2141c7d395c467084
/avatars/fcaa0e4a95796f2ac7000991f1aa19e5.svg
Agus Gunawan
user
agusgun
67628a706562d8e0d34a6ed8
Soo Ye Kim
67628a706562d8e0d34a6ed9
Jihyong Oh
claimed_verified
2024-12-18T12:38:29.484Z
6576b99d58ce19fa1e33eb1d
/avatars/b533e776aa3d95d722b46ef0cd381acd.svg
Jihyong Oh
user
ozbro
67628a706562d8e0d34a6eda
Munchurl Kim
MIVE: New Design and Benchmark for Multi-Instance Video Editing
Recent AI-based video editing has enabled users to edit videos through simple
text prompts, significantly simplifying the editing process. However, recent
zero-shot video editing techniques primarily focus on global or single-object
edits, which can lead to unintended changes in other parts of the video. When
multiple objects require localized edits, existing methods face challenges,
such as unfaithful editing, editing leakage, and lack of suitable evaluation
datasets and metrics. To overcome these limitations, we propose a zero-shot
Multi-Instance Video Editing
framework, called MIVE. MIVE is a general-purpose mask-based framework, not
dedicated to specific objects (e.g., people). MIVE introduces two key modules:
(i) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage and
(ii) Instance-centric Probability Redistribution (IPR) to ensure precise
localization and faithful editing. Additionally, we present our new MIVE
Dataset featuring diverse video scenarios and introduce the Cross-Instance
Accuracy (CIA) Score to evaluate editing leakage in multi-instance video
editing tasks. Our extensive qualitative, quantitative, and user study
evaluations demonstrate that MIVE significantly outperforms recent
state-of-the-art methods in terms of editing faithfulness, accuracy, and
leakage prevention, setting a new benchmark for multi-instance video editing.
The project page is available at https://kaist-viclab.github.io/mive-site/
MIVE is a mask-based framework for zero-shot multi-instance video editing that addresses editing leakage and localization issues using Disentangled Multi-instance Sampling and Instance-centric Probability Redistribution, evaluated with a new dataset and CIA score.
zero-shot mask-based multi-instance editing Disentangled Multi-instance Sampling Instance-centric Probability Redistribution MIVE Dataset Cross-Instance Accuracy editing leakage localization