title: SHINOBI: Shape and Illumination using Neural Object Decomposition via
    BRDF Optimization In-the-wild
summary: We present SHINOBI, an end-to-end framework for the reconstruction of shape,
  material, and illumination from object images captured with varying lighting,
  pose, and background. Inverse rendering of an object based on unconstrained
  image collections is a long-standing challenge in computer vision and graphics
  and requires a joint optimization over shape, radiance, and pose. We show that
  an implicit shape representation based on a multi-resolution hash encoding
  enables faster and robust shape reconstruction with joint camera alignment
  optimization that outperforms prior work. Further, to enable the editing of
  illumination and object reflectance (i.e. material) we jointly optimize BRDF
  and illumination together with the object's shape. Our method is class-agnostic
  and works on in-the-wild image collections of objects to produce relightable 3D
  assets for several use cases such as AR/VR, movies, games, etc. Project page:
  https://shinobi.aengelhardt.com Video:
  https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be
paper_authors._id: 65a9f3dc2582acc636a35ba4
paper_authors.name: Andreas Engelhardt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:28:55.172Z
paper_authors._id: 64ad09cd2d29d3bdcd482bbd
paper_authors.avatarUrl: /avatars/657e02321bc4798bb31eab8b1b9e1c84.svg
paper_authors.fullname: Andreas Engelhardt
paper_authors.type: user
paper_authors.user: andreasengelhardt
paper_authors._id: 65a9f3dc2582acc636a35ba5
paper_authors.name: Amit Raj
paper_authors._id: 65a9f3dc2582acc636a35ba6
paper_authors.name: Mark Boss
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:22:45.638Z
paper_authors._id: 64b7f06fda8017900e893eb4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b7f06fda8017900e893eb4/3VcQFjERXyAjHQFLMFEJt.jpeg
paper_authors.fullname: Mark Boss
paper_authors.type: user
paper_authors.user: mboss
paper_authors._id: 65a9f3dc2582acc636a35ba7
paper_authors.name: Yunzhi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:29:22.998Z
paper_authors._id: 63793aee2cb50dda39d6e73b
paper_authors.avatarUrl: /avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
paper_authors.fullname: Yunzhi Zhang
paper_authors.type: user
paper_authors.user: Yunzhi
paper_authors._id: 65a9f3dc2582acc636a35ba8
paper_authors.name: Abhishek Kar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T08:06:43.112Z
paper_authors._id: 65730b1fa5f2213c55dae1db
paper_authors.avatarUrl: /avatars/cd3bec5588e406bf0c8ba53705ed6123.svg
paper_authors.fullname: Abhishek Kar
paper_authors.type: user
paper_authors.user: akar43
paper_authors._id: 65a9f3dc2582acc636a35ba9
paper_authors.name: Yuanzhen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:29:40.221Z
paper_authors._id: 6334ee5831a2be3938c85639
paper_authors.avatarUrl: /avatars/9821af26a585f54b595ec39f061f2250.svg
paper_authors.fullname: Yuanzhen Li
paper_authors.type: user
paper_authors.user: yzli
paper_authors._id: 65a9f3dc2582acc636a35baa
paper_authors.name: Deqing Sun
paper_authors._id: 65a9f3dc2582acc636a35bab
paper_authors.name: Ricardo Martin Brualla
paper_authors._id: 65a9f3dc2582acc636a35bac
paper_authors.name: Jonathan T. Barron
paper_authors._id: 65a9f3dc2582acc636a35bad
paper_authors.name: Hendrik P. A. Lensch
paper_authors._id: 65a9f3dc2582acc636a35bae
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:30:15.473Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_title: SHINOBI: Shape and Illumination using Neural Object Decomposition via
    BRDF Optimization In-the-wild
paper_summary: We present SHINOBI, an end-to-end framework for the reconstruction of shape,
  material, and illumination from object images captured with varying lighting,
  pose, and background. Inverse rendering of an object based on unconstrained
  image collections is a long-standing challenge in computer vision and graphics
  and requires a joint optimization over shape, radiance, and pose. We show that
  an implicit shape representation based on a multi-resolution hash encoding
  enables faster and robust shape reconstruction with joint camera alignment
  optimization that outperforms prior work. Further, to enable the editing of
  illumination and object reflectance (i.e. material) we jointly optimize BRDF
  and illumination together with the object's shape. Our method is class-agnostic
  and works on in-the-wild image collections of objects to produce relightable 3D
  assets for several use cases such as AR/VR, movies, games, etc. Project page:
  https://shinobi.aengelhardt.com Video:
  https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be
paper_ai_summary: SHINOBI reconstructs 3D shapes, materials, and illumination from unconstrained images using implicit shape representation and joint optimization of BRDF and camera alignment.
paper_ai_keywords: implicit shape representation multi-resolution hash encoding joint camera alignment optimization BRDF inverse rendering 3D assets AR/VR movies games