title: PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
    Observations
summary: Modeling and rendering photorealistic avatars is of crucial importance in
  many applications. Existing methods that build a 3D avatar from visual
  observations, however, struggle to reconstruct clothed humans. We introduce
  PhysAvatar, a novel framework that combines inverse rendering with inverse
  physics to automatically estimate the shape and appearance of a human from
  multi-view video data along with the physical parameters of the fabric of their
  clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
  spatio-temporal mesh tracking as well as a physically based inverse renderer to
  estimate the intrinsic material properties. PhysAvatar integrates a physics
  simulator to estimate the physical parameters of the garments using
  gradient-based optimization in a principled manner. These novel capabilities
  enable PhysAvatar to create high-quality novel-view renderings of avatars
  dressed in loose-fitting clothes under motions and lighting conditions not seen
  in the training data. This marks a significant advancement towards modeling
  photorealistic digital humans using physically based inverse rendering with
  physics in the loop. Our project website is at:
  https://qingqing-zhao.github.io/PhysAvatar
paper_authors._id: 6614ca518cb1d029bac0f88d
paper_authors.name: Yang Zheng
paper_authors._id: 6614ca518cb1d029bac0f88e
paper_authors.name: Qingqing Zhao
paper_authors._id: 6614ca518cb1d029bac0f88f
paper_authors.name: Guandao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-25T06:55:08.036Z
paper_authors._id: 64547ca006728ff79a38e7a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64547ca006728ff79a38e7a5/xfTVQXMSfRv-IbljFpswQ.jpeg
paper_authors.fullname: Guandao Yang
paper_authors.type: user
paper_authors.user: guandao
paper_authors._id: 6614ca518cb1d029bac0f890
paper_authors.name: Wang Yifan
paper_authors._id: 6614ca518cb1d029bac0f891
paper_authors.name: Donglai Xiang
paper_authors._id: 6614ca518cb1d029bac0f892
paper_authors.name: Florian Dubost
paper_authors._id: 6614ca518cb1d029bac0f893
paper_authors.name: Dmitry Lagun
paper_authors._id: 6614ca518cb1d029bac0f894
paper_authors.name: Thabo Beeler
paper_authors._id: 6614ca518cb1d029bac0f895
paper_authors.name: Federico Tombari
paper_authors._id: 6614ca518cb1d029bac0f896
paper_authors.name: Leonidas Guibas
paper_authors._id: 6614ca518cb1d029bac0f897
paper_authors.name: Gordon Wetzstein
paper_title: PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
    Observations
paper_summary: Modeling and rendering photorealistic avatars is of crucial importance in
  many applications. Existing methods that build a 3D avatar from visual
  observations, however, struggle to reconstruct clothed humans. We introduce
  PhysAvatar, a novel framework that combines inverse rendering with inverse
  physics to automatically estimate the shape and appearance of a human from
  multi-view video data along with the physical parameters of the fabric of their
  clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
  spatio-temporal mesh tracking as well as a physically based inverse renderer to
  estimate the intrinsic material properties. PhysAvatar integrates a physics
  simulator to estimate the physical parameters of the garments using
  gradient-based optimization in a principled manner. These novel capabilities
  enable PhysAvatar to create high-quality novel-view renderings of avatars
  dressed in loose-fitting clothes under motions and lighting conditions not seen
  in the training data. This marks a significant advancement towards modeling
  photorealistic digital humans using physically based inverse rendering with
  physics in the loop. Our project website is at:
  https://qingqing-zhao.github.io/PhysAvatar
paper_ai_keywords: mesh-aligned 4D Gaussian spatio-temporal mesh tracking physically based inverse renderer intrinsic material properties physics simulator gradient-based optimization physically based inverse rendering physics in the loop