title: Neural feels with neural fields: Visuo-tactile perception for in-hand
    manipulation
summary: To achieve human-level dexterity, robots must infer spatial awareness from
  multimodal sensing to reason over contact interactions. During in-hand
  manipulation of novel objects, such spatial awareness involves estimating the
  object's pose and shape. The status quo for in-hand perception primarily
  employs vision, and restricts to tracking a priori known objects. Moreover,
  visual occlusion of objects in-hand is imminent during manipulation, preventing
  current systems to push beyond tasks without occlusion. We combine vision and
  touch sensing on a multi-fingered hand to estimate an object's pose and shape
  during in-hand manipulation. Our method, NeuralFeels, encodes object geometry
  by learning a neural field online and jointly tracks it by optimizing a pose
  graph problem. We study multimodal in-hand perception in simulation and the
  real-world, interacting with different objects via a proprioception-driven
  policy. Our experiments show final reconstruction F-scores of 81% and average
  pose drifts of 4.7,mm, further reduced to 2.3,mm with known
  CAD models. Additionally, we observe that under heavy visual occlusion we can
  achieve up to 94% improvements in tracking compared to vision-only methods.
  Our results demonstrate that touch, at the very least, refines and, at the very
  best, disambiguates visual estimates during in-hand manipulation. We release
  our evaluation dataset of 70 experiments, FeelSight, as a step towards
  benchmarking in this domain. Our neural representation driven by multimodal
  sensing can serve as a perception backbone towards advancing robot dexterity.
  Videos can be found on our project website
  https://suddhu.github.io/neural-feels/
paper_authors._id: 6584ecd0f8fae1c1e5b0927f
paper_authors.name: Sudharshan Suresh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:39:14.099Z
paper_authors._id: 645ba7079522edfc8e4fbf19
paper_authors.avatarUrl: /avatars/888e3219d7591accc3a23bba8532273e.svg
paper_authors.fullname: Sudharshan Suresh
paper_authors.type: user
paper_authors.user: suddhu
paper_authors._id: 6584ecd0f8fae1c1e5b09280
paper_authors.name: Haozhi Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:37:57.135Z
paper_authors._id: 658871e7d1331d552be8b457
paper_authors.avatarUrl: /avatars/c323178ad1e0fbdfa5e66dcd4ba357e6.svg
paper_authors.fullname: Haozhi Qi
paper_authors.type: user
paper_authors.user: HaozhiQi
paper_authors._id: 6584ecd0f8fae1c1e5b09281
paper_authors.name: Tingfan Wu
paper_authors._id: 6584ecd0f8fae1c1e5b09282
paper_authors.name: Taosha Fan
paper_authors._id: 6584ecd0f8fae1c1e5b09283
paper_authors.name: Luis Pineda
paper_authors._id: 6584ecd0f8fae1c1e5b09284
paper_authors.name: Mike Lambeta
paper_authors._id: 6584ecd0f8fae1c1e5b09285
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:43:13.318Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_authors._id: 6584ecd0f8fae1c1e5b09286
paper_authors.name: Mrinal Kalakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:42:53.762Z
paper_authors._id: 64cad11dead94891d1e72243
paper_authors.avatarUrl: /avatars/d7faa419ccb0182ad081555491b80d14.svg
paper_authors.fullname: Mrinal Kalakrishnan
paper_authors.type: user
paper_authors.user: kalakris
paper_authors._id: 6584ecd0f8fae1c1e5b09287
paper_authors.name: Roberto Calandra
paper_authors._id: 6584ecd0f8fae1c1e5b09288
paper_authors.name: Michael Kaess
paper_authors._id: 6584ecd0f8fae1c1e5b09289
paper_authors.name: Joseph Ortiz
paper_authors._id: 6584ecd0f8fae1c1e5b0928a
paper_authors.name: Mustafa Mukadam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-02T09:28:09.480Z
paper_authors._id: 65901cd989f1ff04636c896c
paper_authors.avatarUrl: /avatars/64d1d58811f31d9304287699e0879d4b.svg
paper_authors.fullname: Mustafa Mustafa
paper_authors.type: user
paper_authors.user: mukadammh
paper_title: Neural feels with neural fields: Visuo-tactile perception for in-hand
    manipulation
paper_summary: To achieve human-level dexterity, robots must infer spatial awareness from
  multimodal sensing to reason over contact interactions. During in-hand
  manipulation of novel objects, such spatial awareness involves estimating the
  object's pose and shape. The status quo for in-hand perception primarily
  employs vision, and restricts to tracking a priori known objects. Moreover,
  visual occlusion of objects in-hand is imminent during manipulation, preventing
  current systems to push beyond tasks without occlusion. We combine vision and
  touch sensing on a multi-fingered hand to estimate an object's pose and shape
  during in-hand manipulation. Our method, NeuralFeels, encodes object geometry
  by learning a neural field online and jointly tracks it by optimizing a pose
  graph problem. We study multimodal in-hand perception in simulation and the
  real-world, interacting with different objects via a proprioception-driven
  policy. Our experiments show final reconstruction F-scores of 81% and average
  pose drifts of 4.7,mm, further reduced to 2.3,mm with known
  CAD models. Additionally, we observe that under heavy visual occlusion we can
  achieve up to 94% improvements in tracking compared to vision-only methods.
  Our results demonstrate that touch, at the very least, refines and, at the very
  best, disambiguates visual estimates during in-hand manipulation. We release
  our evaluation dataset of 70 experiments, FeelSight, as a step towards
  benchmarking in this domain. Our neural representation driven by multimodal
  sensing can serve as a perception backbone towards advancing robot dexterity.
  Videos can be found on our project website
  https://suddhu.github.io/neural-feels/
paper_ai_summary: NeuralFeels, a multimodal perception method combining vision and touch, enhances spatial awareness and object tracking during in-hand manipulation, especially under visual occlusion.
paper_ai_keywords: neural field pose graph problem proprioception-driven policy F-scores pose drifts visual occlusion multimodal sensing perception backbone