title: TacSL: A Library for Visuotactile Sensor Simulation and Learning
summary: For both humans and robots, the sense of touch, known as tactile sensing, is
  critical for performing contact-rich manipulation tasks. Three key challenges
  in robotic tactile sensing are 1) interpreting sensor signals, 2) generating
  sensor signals in novel scenarios, and 3) learning sensor-based policies. For
  visuotactile sensors, interpretation has been facilitated by their close
  relationship with vision sensors (e.g., RGB cameras). However, generation is
  still difficult, as visuotactile sensors typically involve contact,
  deformation, illumination, and imaging, all of which are expensive to simulate;
  in turn, policy learning has been challenging, as simulation cannot be
  leveraged for large-scale data collection. We present TacSL
  (taxel), a library for GPU-based visuotactile sensor simulation and
  learning. TacSL can be used to simulate visuotactile images and
  extract contact-force distributions over 200times faster than the prior
  state-of-the-art, all within the widely-used Isaac Gym simulator. Furthermore,
  TacSL provides a learning toolkit containing multiple sensor models,
  contact-intensive training environments, and online/offline algorithms that can
  facilitate policy learning for sim-to-real applications. On the algorithmic
  side, we introduce a novel online reinforcement-learning algorithm called
  asymmetric actor-critic distillation (\sysName), designed to effectively and
  efficiently learn tactile-based policies in simulation that can transfer to the
  real world. Finally, we demonstrate the utility of our library and algorithms
  by evaluating the benefits of distillation and multimodal sensing for
  contact-rich manip ulation tasks, and most critically, performing sim-to-real
  transfer. Supplementary videos and results are at
  https://iakinola23.github.io/tacsl/.
paper_authors._id: 66bc2863bfa218d3b155d874
paper_authors.name: Iretiayo Akinola
paper_authors._id: 66bc2863bfa218d3b155d875
paper_authors.name: Jie Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T21:02:53.130Z
paper_authors._id: 66bf785fdccd1164803e0270
paper_authors.avatarUrl: /avatars/e762628f8b8f18cefd6a8ed3a7810ec9.svg
paper_authors.fullname: Jie Xu
paper_authors.type: user
paper_authors.user: xujie7979
paper_authors._id: 66bc2863bfa218d3b155d876
paper_authors.name: Jan Carius
paper_authors._id: 66bc2863bfa218d3b155d877
paper_authors.name: Dieter Fox
paper_authors._id: 66bc2863bfa218d3b155d878
paper_authors.name: Yashraj Narang
paper_title: TacSL: A Library for Visuotactile Sensor Simulation and Learning
paper_summary: For both humans and robots, the sense of touch, known as tactile sensing, is
  critical for performing contact-rich manipulation tasks. Three key challenges
  in robotic tactile sensing are 1) interpreting sensor signals, 2) generating
  sensor signals in novel scenarios, and 3) learning sensor-based policies. For
  visuotactile sensors, interpretation has been facilitated by their close
  relationship with vision sensors (e.g., RGB cameras). However, generation is
  still difficult, as visuotactile sensors typically involve contact,
  deformation, illumination, and imaging, all of which are expensive to simulate;
  in turn, policy learning has been challenging, as simulation cannot be
  leveraged for large-scale data collection. We present TacSL
  (taxel), a library for GPU-based visuotactile sensor simulation and
  learning. TacSL can be used to simulate visuotactile images and
  extract contact-force distributions over 200times faster than the prior
  state-of-the-art, all within the widely-used Isaac Gym simulator. Furthermore,
  TacSL provides a learning toolkit containing multiple sensor models,
  contact-intensive training environments, and online/offline algorithms that can
  facilitate policy learning for sim-to-real applications. On the algorithmic
  side, we introduce a novel online reinforcement-learning algorithm called
  asymmetric actor-critic distillation (\sysName), designed to effectively and
  efficiently learn tactile-based policies in simulation that can transfer to the
  real world. Finally, we demonstrate the utility of our library and algorithms
  by evaluating the benefits of distillation and multimodal sensing for
  contact-rich manip ulation tasks, and most critically, performing sim-to-real
  transfer. Supplementary videos and results are at
  https://iakinola23.github.io/tacsl/.
paper_ai_keywords: visuotactile sensors tactile sensing contact-rich manipulation tasks sensor signals interpretation generation sensor-based policies visuotactile images contact-force distribution Isaac Gym simulator sensor models contact-intensive training environments online/offline algorithms asymmetric actor-critic distillation tactile-based policies sim-to-real applications distillation multimodal sensing