Reconstructing Hand-Held Objects in 3D
Objects manipulated by the hand (i.e., manipulanda) are particularly
challenging to reconstruct from in-the-wild RGB images or videos. Not only does
the hand occlude much of the object, but also the object is often only visible
in a small number of image pixels. At the same time, two strong anchors emerge
in this setting: (1) estimated 3D hands help disambiguate the location and
scale of the object, and (2) the set of manipulanda is small relative to all
possible objects. With these insights in mind, we present a scalable paradigm
for handheld object reconstruction that builds on recent breakthroughs in large
language/vision models and 3D object datasets. Our model, MCC-Hand-Object
(MCC-HO), jointly reconstructs hand and object geometry given a single RGB
image and inferred 3D hand as inputs. Subsequently, we use GPT-4(V) to retrieve
a 3D object model that matches the object in the image and rigidly align the
model to the network-inferred geometry; we call this alignment
Retrieval-Augmented Reconstruction (RAR). Experiments demonstrate that MCC-HO
achieves state-of-the-art performance on lab and Internet datasets, and we show
how RAR can be used to automatically obtain 3D labels for in-the-wild images of
hand-object interactions.
6615fa57b3d7d260010411da
Jane Wu
claimed_verified
2024-04-10T10:58:54.977Z
656542cf39002bd4272b2d2e
https://cdn-avatars.huggingface.co/v1/production/uploads/656542cf39002bd4272b2d2e/G5xYUvG9Npjj5Au2i5y0U.jpeg
Jane Wu
user
janehwu
6615fa57b3d7d260010411db
Georgios Pavlakos
6615fa57b3d7d260010411dc
Georgia Gkioxari
claimed_verified
2025-06-29T10:15:33.965Z
634ededbd049354d7ee4b557
https://cdn-avatars.huggingface.co/v1/production/uploads/634ededbd049354d7ee4b557/XJS3qqpExVDFCKjT0NMi8.jpeg
Georgia Gkioxari
user
gkioxari
6615fa57b3d7d260010411dd
Jitendra Malik
Reconstructing Hand-Held Objects in 3D
Objects manipulated by the hand (i.e., manipulanda) are particularly
challenging to reconstruct from in-the-wild RGB images or videos. Not only does
the hand occlude much of the object, but also the object is often only visible
in a small number of image pixels. At the same time, two strong anchors emerge
in this setting: (1) estimated 3D hands help disambiguate the location and
scale of the object, and (2) the set of manipulanda is small relative to all
possible objects. With these insights in mind, we present a scalable paradigm
for handheld object reconstruction that builds on recent breakthroughs in large
language/vision models and 3D object datasets. Our model, MCC-Hand-Object
(MCC-HO), jointly reconstructs hand and object geometry given a single RGB
image and inferred 3D hand as inputs. Subsequently, we use GPT-4(V) to retrieve
a 3D object model that matches the object in the image and rigidly align the
model to the network-inferred geometry; we call this alignment
Retrieval-Augmented Reconstruction (RAR). Experiments demonstrate that MCC-HO
achieves state-of-the-art performance on lab and Internet datasets, and we show
how RAR can be used to automatically obtain 3D labels for in-the-wild images of
hand-object interactions.
A model named MCC-HO reconstructs handheld objects from single RGB images using 3D hand estimation and GPT-4(V) for 3D object retrieval and alignment, achieving top performance on lab and Internet datasets.
MCC-Hand-Object MCC-HO GPT-4(V) Retrieval-Augmented Reconstruction RAR 3D hand estimation