title: URHand: Universal Relightable Hands
summary: Existing photorealistic relightable hand models require extensive
  identity-specific observations in different views, poses, and illuminations,
  and face challenges in generalizing to natural illuminations and novel
  identities. To bridge this gap, we present URHand, the first universal
  relightable hand model that generalizes across viewpoints, poses,
  illuminations, and identities. Our model allows few-shot personalization using
  images captured with a mobile phone, and is ready to be photorealistically
  rendered under novel illuminations. To simplify the personalization process
  while retaining photorealism, we build a powerful universal relightable prior
  based on neural relighting from multi-view images of hands captured in a light
  stage with hundreds of identities. The key challenge is scaling the
  cross-identity training while maintaining personalized fidelity and sharp
  details without compromising generalization under natural illuminations. To
  this end, we propose a spatially varying linear lighting model as the neural
  renderer that takes physics-inspired shading as input feature. By removing
  non-linear activations and bias, our specifically designed lighting model
  explicitly keeps the linearity of light transport. This enables single-stage
  training from light-stage data while generalizing to real-time rendering under
  arbitrary continuous illuminations across diverse identities. In addition, we
  introduce the joint learning of a physically based model and our neural
  relighting model, which further improves fidelity and generalization. Extensive
  experiments show that our approach achieves superior performance over existing
  methods in terms of both quality and generalizability. We also demonstrate
  quick personalization of URHand from a short phone scan of an unseen identity.
paper_authors._id: 659f5cfd70cf8f1cbbf90e40
paper_authors.name: Zhaoxi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T10:39:37.408Z
paper_authors._id: 62fc8cf7ee999004b5a8b982
paper_authors.avatarUrl: /avatars/6c5dda9e58747054a989f077a078f3dc.svg
paper_authors.fullname: Zhaoxi Chen
paper_authors.type: user
paper_authors.user: FrozenBurning
paper_authors._id: 659f5cfd70cf8f1cbbf90e41
paper_authors.name: Gyeongsik Moon
paper_authors._id: 659f5cfd70cf8f1cbbf90e42
paper_authors.name: Kaiwen Guo
paper_authors._id: 659f5cfd70cf8f1cbbf90e43
paper_authors.name: Chen Cao
paper_authors._id: 659f5cfd70cf8f1cbbf90e44
paper_authors.name: Stanislav Pidhorskyi
paper_authors._id: 659f5cfd70cf8f1cbbf90e45
paper_authors.name: Tomas Simon
paper_authors._id: 659f5cfd70cf8f1cbbf90e46
paper_authors.name: Rohan Joshi
paper_authors._id: 659f5cfd70cf8f1cbbf90e47
paper_authors.name: Yuan Dong
paper_authors._id: 659f5cfd70cf8f1cbbf90e48
paper_authors.name: Yichen Xu
paper_authors._id: 659f5cfd70cf8f1cbbf90e49
paper_authors.name: Bernardo Pires
paper_authors._id: 659f5cfd70cf8f1cbbf90e4a
paper_authors.name: He Wen
paper_authors._id: 659f5cfd70cf8f1cbbf90e4b
paper_authors.name: Lucas Evans
paper_authors._id: 659f5cfd70cf8f1cbbf90e4c
paper_authors.name: Bo Peng
paper_authors._id: 659f5cfd70cf8f1cbbf90e4d
paper_authors.name: Julia Buffalini
paper_authors._id: 659f5cfd70cf8f1cbbf90e4e
paper_authors.name: Autumn Trimble
paper_authors._id: 659f5cfd70cf8f1cbbf90e4f
paper_authors.name: Kevyn McPhail
paper_authors._id: 659f5cfd70cf8f1cbbf90e50
paper_authors.name: Melissa Schoeller
paper_authors._id: 659f5cfd70cf8f1cbbf90e51
paper_authors.name: Shoou-I Yu
paper_authors._id: 659f5cfd70cf8f1cbbf90e52
paper_authors.name: Javier Romero
paper_authors._id: 659f5cfd70cf8f1cbbf90e53
paper_authors.name: Michael Zollhöfer
paper_authors._id: 659f5cfd70cf8f1cbbf90e54
paper_authors.name: Yaser Sheikh
paper_authors._id: 659f5cfd70cf8f1cbbf90e55
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T10:39:16.197Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 659f5cfd70cf8f1cbbf90e56
paper_authors.name: Shunsuke Saito
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:17:03.981Z
paper_authors._id: 630688edd37ce67e0e50e548
paper_authors.avatarUrl: /avatars/1681e465d7649f67f94e1b69d236cb1e.svg
paper_authors.fullname: Shunsuke Saito
paper_authors.type: user
paper_authors.user: psyth
paper_title: URHand: Universal Relightable Hands
paper_summary: Existing photorealistic relightable hand models require extensive
  identity-specific observations in different views, poses, and illuminations,
  and face challenges in generalizing to natural illuminations and novel
  identities. To bridge this gap, we present URHand, the first universal
  relightable hand model that generalizes across viewpoints, poses,
  illuminations, and identities. Our model allows few-shot personalization using
  images captured with a mobile phone, and is ready to be photorealistically
  rendered under novel illuminations. To simplify the personalization process
  while retaining photorealism, we build a powerful universal relightable prior
  based on neural relighting from multi-view images of hands captured in a light
  stage with hundreds of identities. The key challenge is scaling the
  cross-identity training while maintaining personalized fidelity and sharp
  details without compromising generalization under natural illuminations. To
  this end, we propose a spatially varying linear lighting model as the neural
  renderer that takes physics-inspired shading as input feature. By removing
  non-linear activations and bias, our specifically designed lighting model
  explicitly keeps the linearity of light transport. This enables single-stage
  training from light-stage data while generalizing to real-time rendering under
  arbitrary continuous illuminations across diverse identities. In addition, we
  introduce the joint learning of a physically based model and our neural
  relighting model, which further improves fidelity and generalization. Extensive
  experiments show that our approach achieves superior performance over existing
  methods in terms of both quality and generalizability. We also demonstrate
  quick personalization of URHand from a short phone scan of an unseen identity.
paper_ai_summary: URHand is a universal relightable hand model that generalizes across different identities, viewpoints, and illuminations, using a spatially varying linear lighting model and joint learning of a physically based model for high-quality, real-time rendering.
paper_ai_keywords: neural relighting light stage spatially varying linear lighting model physics-inspired shading single-stage training physically based model joint learning