VIA: A Spatiotemporal Video Adaptation Framework for Global and Local
  Video Editing
Video editing stands as a cornerstone of digital media, from entertainment
and education to professional communication. However, previous methods often
overlook the necessity of comprehensively understanding both global and local
contexts, leading to inaccurate and inconsistency edits in the spatiotemporal
dimension, especially for long videos. In this paper, we introduce VIA, a
unified spatiotemporal VIdeo Adaptation framework for global and local video
editing, pushing the limits of consistently editing minute-long videos. First,
to ensure local consistency within individual frames, the foundation of VIA is
a novel test-time editing adaptation method, which adapts a pre-trained image
editing model for improving consistency between potential editing directions
and the text instruction, and adapts masked latent variables for precise local
control. Furthermore, to maintain global consistency over the video sequence,
we introduce spatiotemporal adaptation that adapts consistent attention
variables in key frames and strategically applies them across the whole
sequence to realize the editing effects. Extensive experiments demonstrate
that, compared to baseline methods, our VIA approach produces edits that are
more faithful to the source videos, more coherent in the spatiotemporal
context, and more precise in local control. More importantly, we show that VIA
can achieve consistent long video editing in minutes, unlocking the potentials
for advanced video editing tasks over long video sequences.
66731088881ffb63ca260483
Jing Gu
admin_assigned
2024-06-20T09:37:09.202Z
634781fae5c0717e6739bc59
/avatars/2b6d2d0625741b28f326459acad55b69.svg
Jing Gu
user
jinggu
66731088881ffb63ca260484
Yuwei Fang
admin_assigned
2024-06-20T09:37:42.492Z
6406d1dcc3ab325efa9db198
/avatars/781f3cd455596f7cffa0906cc487f79b.svg
Yuwei Fang
user
studyfang
66731088881ffb63ca260485
Ivan Skorokhodov
admin_assigned
2024-06-20T09:37:52.168Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
66731088881ffb63ca260486
Peter Wonka
66731088881ffb63ca260487
Xinya Du
claimed_verified
2024-09-16T07:03:31.181Z
66c4154fb83a7e94d588ade3
/avatars/2f1c8a3feb81be628e36eaa198450704.svg
Xinya Du
user
xinyadu
66731088881ffb63ca260488
Sergey Tulyakov
66731088881ffb63ca260489
Xin Eric Wang
claimed_verified
2024-06-20T07:34:17.594Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
VIA: A Spatiotemporal Video Adaptation Framework for Global and Local
  Video Editing
Video editing stands as a cornerstone of digital media, from entertainment
and education to professional communication. However, previous methods often
overlook the necessity of comprehensively understanding both global and local
contexts, leading to inaccurate and inconsistency edits in the spatiotemporal
dimension, especially for long videos. In this paper, we introduce VIA, a
unified spatiotemporal VIdeo Adaptation framework for global and local video
editing, pushing the limits of consistently editing minute-long videos. First,
to ensure local consistency within individual frames, the foundation of VIA is
a novel test-time editing adaptation method, which adapts a pre-trained image
editing model for improving consistency between potential editing directions
and the text instruction, and adapts masked latent variables for precise local
control. Furthermore, to maintain global consistency over the video sequence,
we introduce spatiotemporal adaptation that adapts consistent attention
variables in key frames and strategically applies them across the whole
sequence to realize the editing effects. Extensive experiments demonstrate
that, compared to baseline methods, our VIA approach produces edits that are
more faithful to the source videos, more coherent in the spatiotemporal
context, and more precise in local control. More importantly, we show that VIA
can achieve consistent long video editing in minutes, unlocking the potentials
for advanced video editing tasks over long video sequences.
VIA, a unified spatiotemporal video editing framework, enhances consistency and precision in global and local editing of long videos.
spatiotemporal adaptation test-time editing adaptation pre-trained image editing model masked latent variables consistent attention variables vide editing spatiotemporal context local control global consistency