title: Object-Centric Diffusion for Efficient Video Editing
summary: Diffusion-based video editing have reached impressive quality and can
  transform either the global style, local structure, and attributes of given
  video inputs, following textual edit prompts. However, such solutions typically
  incur heavy memory and computational costs to generate temporally-coherent
  frames, either in the form of diffusion inversion and/or cross-frame attention.
  In this paper, we conduct an analysis of such inefficiencies, and suggest
  simple yet effective modifications that allow significant speed-ups whilst
  maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as
  OCD, to further reduce latency by allocating computations more towards
  foreground edited regions that are arguably more important for perceptual
  quality. We achieve this by two novel proposals: i) Object-Centric Sampling,
  decoupling the diffusion steps spent on salient regions or background,
  allocating most of the model capacity to the former, and ii) Object-Centric 3D
  Token Merging, which reduces cost of cross-frame attention by fusing redundant
  tokens in unimportant background regions. Both techniques are readily
  applicable to a given video editing model without retraining, and can
  drastically reduce its memory and computational cost. We evaluate our proposals
  on inversion-based and control-signal-based editing pipelines, and show a
  latency reduction up to 10x for a comparable synthesis quality.
paper_authors._id: 65a0ba66c5770b27aebc4d19
paper_authors.name: Kumara Kahatapitiya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-12T13:42:11.048Z
paper_authors._id: 65a12ec4c9873890d15c4ab9
paper_authors.avatarUrl: /avatars/ce4d46f575ba757f78eabdb25b394171.svg
paper_authors.fullname: Kumara Kahatapitiya
paper_authors.type: user
paper_authors.user: kumarak
paper_authors._id: 65a0ba66c5770b27aebc4d1a
paper_authors.name: Adil Karjauv
paper_authors._id: 65a0ba66c5770b27aebc4d1b
paper_authors.name: Davide Abati
paper_authors._id: 65a0ba66c5770b27aebc4d1c
paper_authors.name: Fatih Porikli
paper_authors._id: 65a0ba66c5770b27aebc4d1d
paper_authors.name: Yuki M. Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:12:08.437Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_authors._id: 65a0ba66c5770b27aebc4d1e
paper_authors.name: Amirhossein Habibian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-12T10:34:12.576Z
paper_authors._id: 628372aa09aa80237c6d2044
paper_authors.avatarUrl: /avatars/c1e34dee217d73f5539a0807adc12c80.svg
paper_authors.fullname: Amir Habibian
paper_authors.type: user
paper_authors.user: habibian
paper_title: Object-Centric Diffusion for Efficient Video Editing
paper_summary: Diffusion-based video editing have reached impressive quality and can
  transform either the global style, local structure, and attributes of given
  video inputs, following textual edit prompts. However, such solutions typically
  incur heavy memory and computational costs to generate temporally-coherent
  frames, either in the form of diffusion inversion and/or cross-frame attention.
  In this paper, we conduct an analysis of such inefficiencies, and suggest
  simple yet effective modifications that allow significant speed-ups whilst
  maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as
  OCD, to further reduce latency by allocating computations more towards
  foreground edited regions that are arguably more important for perceptual
  quality. We achieve this by two novel proposals: i) Object-Centric Sampling,
  decoupling the diffusion steps spent on salient regions or background,
  allocating most of the model capacity to the former, and ii) Object-Centric 3D
  Token Merging, which reduces cost of cross-frame attention by fusing redundant
  tokens in unimportant background regions. Both techniques are readily
  applicable to a given video editing model without retraining, and can
  drastically reduce its memory and computational cost. We evaluate our proposals
  on inversion-based and control-signal-based editing pipelines, and show a
  latency reduction up to 10x for a comparable synthesis quality.
paper_ai_summary: Object-Centric Diffusion reduces memory and computational cost in diffusion-based video editing by focusing on salient regions and merging redundant tokens in the background, achieving up to a 10x latency reduction with comparable quality.
paper_ai_keywords: diffusion-based video editing textual edit prompts diffusion inversion cross-frame attention Object-Centric Diffusion OCD Object-Centric Sampling Object-Centric 3D Token Merging