title: Reconstructing Animatable Categories from Videos
summary: Building animatable 3D models is challenging due to the need for 3D scans,
  laborious registration, and manual rigging, which are difficult to scale to
  arbitrary categories. Recently, differentiable rendering provides a pathway to
  obtain high-quality 3D models from monocular videos, but these are limited to
  rigid categories or single instances. We present RAC that builds category 3D
  models from monocular videos while disentangling variations over instances and
  motion over time. Three key ideas are introduced to solve this problem: (1)
  specializing a skeleton to instances via optimization, (2) a method for latent
  space regularization that encourages shared structure across a category while
  maintaining instance details, and (3) using 3D background models to disentangle
  objects from the background. We show that 3D models of humans, cats, and dogs
  can be learned from 50-100 internet videos.
paper_authors._id: 645c64d8fbdc26a155fd4f6e
paper_authors.name: Gengshan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:03:40.546Z
paper_authors._id: 62c0597543d4858f821bc87a
paper_authors.avatarUrl: /avatars/ce9d28c08a5c9e8312a754714bae4634.svg
paper_authors.fullname: Gengshan Yang
paper_authors.type: user
paper_authors.user: gengshan-y
paper_authors._id: 645c64d8fbdc26a155fd4f6f
paper_authors.name: Chaoyang Wang
paper_authors._id: 645c64d8fbdc26a155fd4f70
paper_authors.name: N Dinesh Reddy
paper_authors._id: 645c64d8fbdc26a155fd4f71
paper_authors.name: Deva Ramanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:03:49.674Z
paper_authors._id: 6337151b0267ebcf02640eb6
paper_authors.avatarUrl: /avatars/14a723cafc5587043bdfb19304fc202d.svg
paper_authors.fullname: Deva Ramanan
paper_authors.type: user
paper_authors.user: devakramanan
paper_title: Reconstructing Animatable Categories from Videos
paper_summary: Building animatable 3D models is challenging due to the need for 3D scans,
  laborious registration, and manual rigging, which are difficult to scale to
  arbitrary categories. Recently, differentiable rendering provides a pathway to
  obtain high-quality 3D models from monocular videos, but these are limited to
  rigid categories or single instances. We present RAC that builds category 3D
  models from monocular videos while disentangling variations over instances and
  motion over time. Three key ideas are introduced to solve this problem: (1)
  specializing a skeleton to instances via optimization, (2) a method for latent
  space regularization that encourages shared structure across a category while
  maintaining instance details, and (3) using 3D background models to disentangle
  objects from the background. We show that 3D models of humans, cats, and dogs
  can be learned from 50-100 internet videos.
paper_ai_summary: RAC learns animatable 3D models from monocular videos by optimizing instance-specific skeletons, regularizing latent spaces, and disentangling objects from backgrounds, enabling category-wide models.
paper_ai_keywords: differentiable rendering monocular videos category 3D models variation disentangling motion disentangling skeleton optimization latent space regularization 3D background models