title: PEEKABOO: Interactive Video Generation via Masked-Diffusion
summary: Recently there has been a lot of progress in text-to-video generation, with
  state-of-the-art models being capable of generating high quality, realistic
  videos. However, these models lack the capability for users to interactively
  control and generate videos, which can potentially unlock new areas of
  application. As a first step towards this goal, we tackle the problem of
  endowing diffusion-based video generation models with interactive
  spatio-temporal control over their output. To this end, we take inspiration
  from the recent advances in segmentation literature to propose a novel
  spatio-temporal masked attention module - Peekaboo. This module is a
  training-free, no-inference-overhead addition to off-the-shelf video generation
  models which enables spatio-temporal control. We also propose an evaluation
  benchmark for the interactive video generation task. Through extensive
  qualitative and quantitative evaluation, we establish that Peekaboo enables
  control video generation and even obtains a gain of upto 3.8x in mIoU over
  baseline models.
paper_authors._id: 6579225f5a1c60302fd1ea0f
paper_authors.name: Yash Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-14T10:34:46.985Z
paper_authors._id: 629c38f8ba19b79d354d009a
paper_authors.avatarUrl: /avatars/2b37db99a4e3a2d6b86ddcbde7e2f97b.svg
paper_authors.fullname: Yash Jain
paper_authors.type: user
paper_authors.user: jingalala
paper_authors._id: 6579225f5a1c60302fd1ea10
paper_authors.name: Anshul Nasery
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T08:32:49.032Z
paper_authors._id: 65762f34ca03b6c514068e1e
paper_authors.avatarUrl: /avatars/2cd0a84e67ba9bad9fa969e7bd3ca886.svg
paper_authors.fullname: Anshul Nasery
paper_authors.type: user
paper_authors.user: anshuln
paper_authors._id: 6579225f5a1c60302fd1ea11
paper_authors.name: Vibhav Vineet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T10:55:36.620Z
paper_authors._id: 63c8527becdb7c9fdd9cacc6
paper_authors.avatarUrl: /avatars/c8a3f5e1e5159ae5ead41bd9fc2b9b34.svg
paper_authors.fullname: Vibhav Vineet
paper_authors.type: user
paper_authors.user: vibhav-vineet
paper_authors._id: 6579225f5a1c60302fd1ea12
paper_authors.name: Harkirat Behl
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-14T10:34:49.711Z
paper_authors._id: 63c6aa9e656e7822e2359d9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c6aa9e656e7822e2359d9c/4rn_faJ23m_a6bevXEDG_.jpeg
paper_authors.fullname: Harkirat Behl
paper_authors.type: user
paper_authors.user: Harkirat
paper_title: PEEKABOO: Interactive Video Generation via Masked-Diffusion
paper_summary: Recently there has been a lot of progress in text-to-video generation, with
  state-of-the-art models being capable of generating high quality, realistic
  videos. However, these models lack the capability for users to interactively
  control and generate videos, which can potentially unlock new areas of
  application. As a first step towards this goal, we tackle the problem of
  endowing diffusion-based video generation models with interactive
  spatio-temporal control over their output. To this end, we take inspiration
  from the recent advances in segmentation literature to propose a novel
  spatio-temporal masked attention module - Peekaboo. This module is a
  training-free, no-inference-overhead addition to off-the-shelf video generation
  models which enables spatio-temporal control. We also propose an evaluation
  benchmark for the interactive video generation task. Through extensive
  qualitative and quantitative evaluation, we establish that Peekaboo enables
  control video generation and even obtains a gain of upto 3.8x in mIoU over
  baseline models.
paper_ai_summary: Peekaboo, a novel spatio-temporal masked attention module, enhances diffusion-based video generation models with interactive control, outperforming baselines in quality.
paper_ai_keywords: diffusion-based video generation models spatio-temporal masked attention module Peekaboo mIoU