Object-Centric Diffusion for Efficient Video Editing
Diffusion-based video editing have reached impressive quality and can
transform either the global style, local structure, and attributes of given
video inputs, following textual edit prompts. However, such solutions typically
incur heavy memory and computational costs to generate temporally-coherent
frames, either in the form of diffusion inversion and/or cross-frame attention.
In this paper, we conduct an analysis of such inefficiencies, and suggest
simple yet effective modifications that allow significant speed-ups whilst
maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as
OCD, to further reduce latency by allocating computations more towards
foreground edited regions that are arguably more important for perceptual
quality. We achieve this by two novel proposals: i) Object-Centric Sampling,
decoupling the diffusion steps spent on salient regions or background,
allocating most of the model capacity to the former, and ii) Object-Centric 3D
Token Merging, which reduces cost of cross-frame attention by fusing redundant
tokens in unimportant background regions. Both techniques are readily
applicable to a given video editing model without retraining, and can
drastically reduce its memory and computational cost. We evaluate our proposals
on inversion-based and control-signal-based editing pipelines, and show a
latency reduction up to 10x for a comparable synthesis quality.
65a0ba66c5770b27aebc4d19
Kumara Kahatapitiya
claimed_verified
2024-01-12T13:42:11.048Z
65a12ec4c9873890d15c4ab9
/avatars/ce4d46f575ba757f78eabdb25b394171.svg
Kumara Kahatapitiya
user
kumarak
65a0ba66c5770b27aebc4d1a
Adil Karjauv
65a0ba66c5770b27aebc4d1b
Davide Abati
65a0ba66c5770b27aebc4d1c
Fatih Porikli
65a0ba66c5770b27aebc4d1d
Yuki M. Asano
claimed_verified
2024-10-15T09:12:08.437Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
65a0ba66c5770b27aebc4d1e
Amirhossein Habibian
claimed_verified
2024-01-12T10:34:12.576Z
628372aa09aa80237c6d2044
/avatars/c1e34dee217d73f5539a0807adc12c80.svg
Amir Habibian
user
habibian
Object-Centric Diffusion for Efficient Video Editing
Diffusion-based video editing have reached impressive quality and can
transform either the global style, local structure, and attributes of given
video inputs, following textual edit prompts. However, such solutions typically
incur heavy memory and computational costs to generate temporally-coherent
frames, either in the form of diffusion inversion and/or cross-frame attention.
In this paper, we conduct an analysis of such inefficiencies, and suggest
simple yet effective modifications that allow significant speed-ups whilst
maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as
OCD, to further reduce latency by allocating computations more towards
foreground edited regions that are arguably more important for perceptual
quality. We achieve this by two novel proposals: i) Object-Centric Sampling,
decoupling the diffusion steps spent on salient regions or background,
allocating most of the model capacity to the former, and ii) Object-Centric 3D
Token Merging, which reduces cost of cross-frame attention by fusing redundant
tokens in unimportant background regions. Both techniques are readily
applicable to a given video editing model without retraining, and can
drastically reduce its memory and computational cost. We evaluate our proposals
on inversion-based and control-signal-based editing pipelines, and show a
latency reduction up to 10x for a comparable synthesis quality.
Object-Centric Diffusion reduces memory and computational cost in diffusion-based video editing by focusing on salient regions and merging redundant tokens in the background, achieving up to a 10x latency reduction with comparable quality.
diffusion-based video editing textual edit prompts diffusion inversion cross-frame attention Object-Centric Diffusion OCD Object-Centric Sampling Object-Centric 3D Token Merging