title: ActAnywhere: Subject-Aware Video Background Generation
summary: Generating video background that tailors to foreground subject motion is an
  important problem for the movie industry and visual effects community. This
  task involves synthesizing background that aligns with the motion and
  appearance of the foreground subject, while also complies with the artist's
  creative intention. We introduce ActAnywhere, a generative model that automates
  this process which traditionally requires tedious manual efforts. Our model
  leverages the power of large-scale video diffusion models, and is specifically
  tailored for this task. ActAnywhere takes a sequence of foreground subject
  segmentation as input and an image that describes the desired scene as
  condition, to produce a coherent video with realistic foreground-background
  interactions while adhering to the condition frame. We train our model on a
  large-scale dataset of human-scene interaction videos. Extensive evaluations
  demonstrate the superior performance of our model, significantly outperforming
  baselines. Moreover, we show that ActAnywhere generalizes to diverse
  out-of-distribution samples, including non-human subjects. Please visit our
  project webpage at https://actanywhere.github.io.
paper_authors._id: 65ade2dbfd4261f53104ec41
paper_authors.name: Boxiao Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-22T13:13:11.788Z
paper_authors._id: 64b5c10904fa6584c0464be2
paper_authors.avatarUrl: /avatars/13398577c6689fc3340c6c4c40cb2815.svg
paper_authors.fullname: Boxiao Pan
paper_authors.type: user
paper_authors.user: bpan01
paper_authors._id: 65ade2dbfd4261f53104ec42
paper_authors.name: Zhan Xu
paper_authors._id: 65ade2dbfd4261f53104ec43
paper_authors.name: Chun-Hao Paul Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-22T13:14:04.766Z
paper_authors._id: 6318069a33a4e2b5d3b47fe5
paper_authors.avatarUrl: /avatars/c9a9818e64186746557466cc7a6d57b8.svg
paper_authors.fullname: CHUN-HAO  HUANG
paper_authors.type: user
paper_authors.user: ccool2486
paper_authors._id: 65ade2dbfd4261f53104ec44
paper_authors.name: Krishna Kumar Singh
paper_authors._id: 65ade2dbfd4261f53104ec45
paper_authors.name: Yang Zhou
paper_authors._id: 65ade2dbfd4261f53104ec46
paper_authors.name: Leonidas J. Guibas
paper_authors._id: 65ade2dbfd4261f53104ec47
paper_authors.name: Jimei Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T21:11:24.870Z
paper_authors._id: 65ae93c53e82498d4cdaa428
paper_authors.avatarUrl: /avatars/b03221ed402247f6e7a80efef834574f.svg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: Jimei
paper_title: ActAnywhere: Subject-Aware Video Background Generation
paper_summary: Generating video background that tailors to foreground subject motion is an
  important problem for the movie industry and visual effects community. This
  task involves synthesizing background that aligns with the motion and
  appearance of the foreground subject, while also complies with the artist's
  creative intention. We introduce ActAnywhere, a generative model that automates
  this process which traditionally requires tedious manual efforts. Our model
  leverages the power of large-scale video diffusion models, and is specifically
  tailored for this task. ActAnywhere takes a sequence of foreground subject
  segmentation as input and an image that describes the desired scene as
  condition, to produce a coherent video with realistic foreground-background
  interactions while adhering to the condition frame. We train our model on a
  large-scale dataset of human-scene interaction videos. Extensive evaluations
  demonstrate the superior performance of our model, significantly outperforming
  baselines. Moreover, we show that ActAnywhere generalizes to diverse
  out-of-distribution samples, including non-human subjects. Please visit our
  project webpage at https://actanywhere.github.io.
paper_ai_summary: ActAnywhere, a generative model utilizing video diffusion, automates coherent background generation for videos, aligning with foreground motion and artistic conditions.
paper_ai_keywords: generative model video diffusion models foreground subject segmentation condition frame foreground-background interactions human-scene interaction videos