DECO: Dense Estimation of 3D Human-Scene Contact In The Wild
Understanding how humans use physical contact to interact with the world is
key to enabling human-centric artificial intelligence. While inferring 3D
contact is crucial for modeling realistic and physically-plausible human-object
interactions, existing methods either focus on 2D, consider body joints rather
than the surface, use coarse 3D body regions, or do not generalize to
in-the-wild images. In contrast, we focus on inferring dense, 3D contact
between the full body surface and objects in arbitrary images. To achieve this,
we first collect DAMON, a new dataset containing dense vertex-level contact
annotations paired with RGB images containing complex human-object and
human-scene contact. Second, we train DECO, a novel 3D contact detector that
uses both body-part-driven and scene-context-driven attention to estimate
vertex-level contact on the SMPL body. DECO builds on the insight that human
observers recognize contact by reasoning about the contacting body parts, their
proximity to scene objects, and the surrounding scene context. We perform
extensive evaluations of our detector on DAMON as well as on the RICH and
BEHAVE datasets. We significantly outperform existing SOTA methods across all
benchmarks. We also show qualitatively that DECO generalizes well to diverse
and challenging real-world human interactions in natural images. The code,
data, and models are available at https://deco.is.tue.mpg.de.
65151acef624f38acb2625c0
Shashank Tripathi
admin_assigned
2023-09-29T07:03:48.529Z
65156f9c71c52d030afed8ed
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xRvgF_QigDV5POLrXdVZk.png
Shashank Tripathi
user
sha2nkt
65151acef624f38acb2625c1
Agniv Chatterjee
admin_assigned
2023-09-28T09:17:05.749Z
64d77df9505306fcd2da8d5b
/avatars/9053d3330d916c08c4ebe3a0a590e64e.svg
Agniv Chatterjee
user
ac5113
65151acef624f38acb2625c2
Jean-Claude Passy
65151acef624f38acb2625c3
Hongwei Yi
admin_assigned
2023-09-28T09:17:20.823Z
6438d307d221ff12edb1e47e
/avatars/638b20e9f7deec235aa2dcc56bc7ecb7.svg
Hongwei Yi
user
chairmenyi
65151acef624f38acb2625c4
Dimitrios Tzionas
claimed_verified
2023-09-28T10:36:19.783Z
6515533dee24d3b4e3a66f01
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jh-1CnLgDgUlkEYmcHaId.jpeg
Dimitrios Tzionas
user
dimtzionas
65151acef624f38acb2625c5
Michael J. Black
admin_assigned
2023-09-28T09:17:33.417Z
649fbababa7a05e3626546c8
/avatars/a9aa4b4914649527094fc1b12779e014.svg
Michael J. Black
user
MichaelJBlack
DECO: Dense Estimation of 3D Human-Scene Contact In The Wild
Understanding how humans use physical contact to interact with the world is
key to enabling human-centric artificial intelligence. While inferring 3D
contact is crucial for modeling realistic and physically-plausible human-object
interactions, existing methods either focus on 2D, consider body joints rather
than the surface, use coarse 3D body regions, or do not generalize to
in-the-wild images. In contrast, we focus on inferring dense, 3D contact
between the full body surface and objects in arbitrary images. To achieve this,
we first collect DAMON, a new dataset containing dense vertex-level contact
annotations paired with RGB images containing complex human-object and
human-scene contact. Second, we train DECO, a novel 3D contact detector that
uses both body-part-driven and scene-context-driven attention to estimate
vertex-level contact on the SMPL body. DECO builds on the insight that human
observers recognize contact by reasoning about the contacting body parts, their
proximity to scene objects, and the surrounding scene context. We perform
extensive evaluations of our detector on DAMON as well as on the RICH and
BEHAVE datasets. We significantly outperform existing SOTA methods across all
benchmarks. We also show qualitatively that DECO generalizes well to diverse
and challenging real-world human interactions in natural images. The code,
data, and models are available at https://deco.is.tue.mpg.de.
A novel 3D contact detector, DECO, infers dense, vertex-level contact between the full body surface and objects in arbitrary images using body-part-driven and scene-context-driven attention, outperforming existing methods across benchmarks.
DAMON dataset vertex-level contact 3D contact detector body-part-driven attention scene-context-driven attention SMPL body DECO dense contact inferring 3D contact RICH dataset BEHAVE dataset