title: Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for
    Reconstructing Challenging Surfaces
summary: Touch and vision go hand in hand, mutually enhancing our ability to
  understand the world. From a research perspective, the problem of mixing touch
  and vision is underexplored and presents interesting challenges. To this end,
  we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data
  (local depth maps) with multi-view vision data to achieve surface
  reconstruction and novel view synthesis. Our method optimises 3D Gaussian
  primitives to accurately model the object's geometry at points of contact. By
  creating a framework that decreases the transmittance at touch locations, we
  achieve a refined surface reconstruction, ensuring a uniformly smooth depth
  map. Touch is particularly useful when considering non-Lambertian objects (e.g.
  shiny or reflective surfaces) since contemporary methods tend to fail to
  reconstruct with fidelity specular highlights. By combining vision and tactile
  sensing, we achieve more accurate geometry reconstructions with fewer images
  than prior methods. We conduct evaluation on objects with glossy and reflective
  surfaces and demonstrate the effectiveness of our approach, offering
  significant improvements in reconstruction quality.
paper_authors._id: 660a2a29e29bac7275f565db
paper_authors.name: Mauro Comi
paper_authors._id: 660a2a29e29bac7275f565dc
paper_authors.name: Alessio Tonioni
paper_authors._id: 660a2a29e29bac7275f565dd
paper_authors.name: Max Yang
paper_authors._id: 660a2a29e29bac7275f565de
paper_authors.name: Jonathan Tremblay
paper_authors._id: 660a2a29e29bac7275f565df
paper_authors.name: Valts Blukis
paper_authors._id: 660a2a29e29bac7275f565e0
paper_authors.name: Yijiong Lin
paper_authors._id: 660a2a29e29bac7275f565e1
paper_authors.name: Nathan F. Lepora
paper_authors._id: 660a2a29e29bac7275f565e2
paper_authors.name: Laurence Aitchison
paper_title: Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for
    Reconstructing Challenging Surfaces
paper_summary: Touch and vision go hand in hand, mutually enhancing our ability to
  understand the world. From a research perspective, the problem of mixing touch
  and vision is underexplored and presents interesting challenges. To this end,
  we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data
  (local depth maps) with multi-view vision data to achieve surface
  reconstruction and novel view synthesis. Our method optimises 3D Gaussian
  primitives to accurately model the object's geometry at points of contact. By
  creating a framework that decreases the transmittance at touch locations, we
  achieve a refined surface reconstruction, ensuring a uniformly smooth depth
  map. Touch is particularly useful when considering non-Lambertian objects (e.g.
  shiny or reflective surfaces) since contemporary methods tend to fail to
  reconstruct with fidelity specular highlights. By combining vision and tactile
  sensing, we achieve more accurate geometry reconstructions with fewer images
  than prior methods. We conduct evaluation on objects with glossy and reflective
  surfaces and demonstrate the effectiveness of our approach, offering
  significant improvements in reconstruction quality.
paper_ai_summary: A novel approach combining touch and vision data improves surface reconstruction and novel view synthesis, especially for non-Lambertian objects, using optimized 3D Gaussian primitives.
paper_ai_keywords: 3D Gaussian primitives