SHINOBI: Shape and Illumination using Neural Object Decomposition via
  BRDF Optimization In-the-wild
We present SHINOBI, an end-to-end framework for the reconstruction of shape,
material, and illumination from object images captured with varying lighting,
pose, and background. Inverse rendering of an object based on unconstrained
image collections is a long-standing challenge in computer vision and graphics
and requires a joint optimization over shape, radiance, and pose. We show that
an implicit shape representation based on a multi-resolution hash encoding
enables faster and robust shape reconstruction with joint camera alignment
optimization that outperforms prior work. Further, to enable the editing of
illumination and object reflectance (i.e. material) we jointly optimize BRDF
and illumination together with the object's shape. Our method is class-agnostic
and works on in-the-wild image collections of objects to produce relightable 3D
assets for several use cases such as AR/VR, movies, games, etc. Project page:
https://shinobi.aengelhardt.com Video:
https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be
65a9f3dc2582acc636a35ba4
Andreas Engelhardt
admin_assigned
2024-01-19T13:28:55.172Z
64ad09cd2d29d3bdcd482bbd
/avatars/657e02321bc4798bb31eab8b1b9e1c84.svg
Andreas Engelhardt
user
andreasengelhardt
65a9f3dc2582acc636a35ba5
Amit Raj
65a9f3dc2582acc636a35ba6
Mark Boss
claimed_verified
2024-01-19T09:22:45.638Z
64b7f06fda8017900e893eb4
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7f06fda8017900e893eb4/3VcQFjERXyAjHQFLMFEJt.jpeg
Mark Boss
user
mboss
65a9f3dc2582acc636a35ba7
Yunzhi Zhang
admin_assigned
2024-01-19T13:29:22.998Z
63793aee2cb50dda39d6e73b
/avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
Yunzhi Zhang
user
Yunzhi
65a9f3dc2582acc636a35ba8
Abhishek Kar
claimed_verified
2024-01-30T08:06:43.112Z
65730b1fa5f2213c55dae1db
/avatars/cd3bec5588e406bf0c8ba53705ed6123.svg
Abhishek Kar
user
akar43
65a9f3dc2582acc636a35ba9
Yuanzhen Li
admin_assigned
2024-01-19T13:29:40.221Z
6334ee5831a2be3938c85639
/avatars/9821af26a585f54b595ec39f061f2250.svg
Yuanzhen Li
user
yzli
65a9f3dc2582acc636a35baa
Deqing Sun
65a9f3dc2582acc636a35bab
Ricardo Martin Brualla
65a9f3dc2582acc636a35bac
Jonathan T. Barron
65a9f3dc2582acc636a35bad
Hendrik P. A. Lensch
65a9f3dc2582acc636a35bae
Varun Jampani
admin_assigned
2024-01-19T13:30:15.473Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
SHINOBI: Shape and Illumination using Neural Object Decomposition via
  BRDF Optimization In-the-wild
We present SHINOBI, an end-to-end framework for the reconstruction of shape,
material, and illumination from object images captured with varying lighting,
pose, and background. Inverse rendering of an object based on unconstrained
image collections is a long-standing challenge in computer vision and graphics
and requires a joint optimization over shape, radiance, and pose. We show that
an implicit shape representation based on a multi-resolution hash encoding
enables faster and robust shape reconstruction with joint camera alignment
optimization that outperforms prior work. Further, to enable the editing of
illumination and object reflectance (i.e. material) we jointly optimize BRDF
and illumination together with the object's shape. Our method is class-agnostic
and works on in-the-wild image collections of objects to produce relightable 3D
assets for several use cases such as AR/VR, movies, games, etc. Project page:
https://shinobi.aengelhardt.com Video:
https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be
SHINOBI reconstructs 3D shapes, materials, and illumination from unconstrained images using implicit shape representation and joint optimization of BRDF and camera alignment.
implicit shape representation multi-resolution hash encoding joint camera alignment optimization BRDF inverse rendering 3D assets AR/VR movies games