title: Reconstructing Hand-Held Objects in 3D
summary: Objects manipulated by the hand (i.e., manipulanda) are particularly
  challenging to reconstruct from in-the-wild RGB images or videos. Not only does
  the hand occlude much of the object, but also the object is often only visible
  in a small number of image pixels. At the same time, two strong anchors emerge
  in this setting: (1) estimated 3D hands help disambiguate the location and
  scale of the object, and (2) the set of manipulanda is small relative to all
  possible objects. With these insights in mind, we present a scalable paradigm
  for handheld object reconstruction that builds on recent breakthroughs in large
  language/vision models and 3D object datasets. Our model, MCC-Hand-Object
  (MCC-HO), jointly reconstructs hand and object geometry given a single RGB
  image and inferred 3D hand as inputs. Subsequently, we use GPT-4(V) to retrieve
  a 3D object model that matches the object in the image and rigidly align the
  model to the network-inferred geometry; we call this alignment
  Retrieval-Augmented Reconstruction (RAR). Experiments demonstrate that MCC-HO
  achieves state-of-the-art performance on lab and Internet datasets, and we show
  how RAR can be used to automatically obtain 3D labels for in-the-wild images of
  hand-object interactions.
paper_authors._id: 6615fa57b3d7d260010411da
paper_authors.name: Jane Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:58:54.977Z
paper_authors._id: 656542cf39002bd4272b2d2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656542cf39002bd4272b2d2e/G5xYUvG9Npjj5Au2i5y0U.jpeg
paper_authors.fullname: Jane Wu
paper_authors.type: user
paper_authors.user: janehwu
paper_authors._id: 6615fa57b3d7d260010411db
paper_authors.name: Georgios Pavlakos
paper_authors._id: 6615fa57b3d7d260010411dc
paper_authors.name: Georgia Gkioxari
paper_authors._id: 6615fa57b3d7d260010411dd
paper_authors.name: Jitendra Malik
paper_title: Reconstructing Hand-Held Objects in 3D
paper_summary: Objects manipulated by the hand (i.e., manipulanda) are particularly
  challenging to reconstruct from in-the-wild RGB images or videos. Not only does
  the hand occlude much of the object, but also the object is often only visible
  in a small number of image pixels. At the same time, two strong anchors emerge
  in this setting: (1) estimated 3D hands help disambiguate the location and
  scale of the object, and (2) the set of manipulanda is small relative to all
  possible objects. With these insights in mind, we present a scalable paradigm
  for handheld object reconstruction that builds on recent breakthroughs in large
  language/vision models and 3D object datasets. Our model, MCC-Hand-Object
  (MCC-HO), jointly reconstructs hand and object geometry given a single RGB
  image and inferred 3D hand as inputs. Subsequently, we use GPT-4(V) to retrieve
  a 3D object model that matches the object in the image and rigidly align the
  model to the network-inferred geometry; we call this alignment
  Retrieval-Augmented Reconstruction (RAR). Experiments demonstrate that MCC-HO
  achieves state-of-the-art performance on lab and Internet datasets, and we show
  how RAR can be used to automatically obtain 3D labels for in-the-wild images of
  hand-object interactions.
paper_ai_keywords: MCC-Hand-Object (MCC-HO) Retrieval-Augmented Reconstruction (RAR) GPT-4(V) 3D hands 3D object datasets large language/vision models 3D object model hand geometry network-inferred geometry lab datasets Internet datasets