title: A Touch, Vision, and Language Dataset for Multimodal Alignment
summary: Touch is an important sensing modality for humans, but it has not yet been
  incorporated into a multimodal generative language model. This is partially due
  to the difficulty of obtaining natural language labels for tactile data and the
  complexity of aligning tactile readings with both visual observations and
  language descriptions. As a step towards bridging that gap, this work
  introduces a new dataset of 44K in-the-wild vision-touch pairs, with English
  language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V
  (90%). We use this dataset to train a vision-language-aligned tactile encoder
  for open-vocabulary classification and a touch-vision-language (TVL) model for
  text generation using the trained encoder. Results suggest that by
  incorporating touch, the TVL model improves (+29% classification accuracy)
  touch-vision-language alignment over existing models trained on any pair of
  those modalities. Although only a small fraction of the dataset is
  human-labeled, the TVL model demonstrates improved visual-tactile understanding
  over GPT-4V (+12%) and open-source vision-language models (+32%) on a new
  touch-vision understanding benchmark. Code and data:
  https://tactile-vlm.github.io.
paper_authors._id: 65d561795717d720a0c544fc
paper_authors.name: Letian Fu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-21T02:35:38.909Z
paper_authors._id: 64908d6b0c18343a09361b93
paper_authors.avatarUrl: /avatars/2fa44344fe760f68f19de775b76f45b8.svg
paper_authors.fullname: Max (Letian) Fu
paper_authors.type: user
paper_authors.user: mlfu7
paper_authors._id: 65d561795717d720a0c544fd
paper_authors.name: Gaurav Datta
paper_authors._id: 65d561795717d720a0c544fe
paper_authors.name: Huang Huang
paper_authors._id: 65d561795717d720a0c544ff
paper_authors.name: William Chung-Ho Panitch
paper_authors._id: 65d561795717d720a0c54500
paper_authors.name: Jaimyn Drake
paper_authors._id: 65d561795717d720a0c54501
paper_authors.name: Joseph Ortiz
paper_authors._id: 65d561795717d720a0c54502
paper_authors.name: Mustafa Mukadam
paper_authors._id: 65d561795717d720a0c54503
paper_authors.name: Mike Lambeta
paper_authors._id: 65d561795717d720a0c54504
paper_authors.name: Roberto Calandra
paper_authors._id: 65d561795717d720a0c54505
paper_authors.name: Ken Goldberg
paper_title: A Touch, Vision, and Language Dataset for Multimodal Alignment
paper_summary: Touch is an important sensing modality for humans, but it has not yet been
  incorporated into a multimodal generative language model. This is partially due
  to the difficulty of obtaining natural language labels for tactile data and the
  complexity of aligning tactile readings with both visual observations and
  language descriptions. As a step towards bridging that gap, this work
  introduces a new dataset of 44K in-the-wild vision-touch pairs, with English
  language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V
  (90%). We use this dataset to train a vision-language-aligned tactile encoder
  for open-vocabulary classification and a touch-vision-language (TVL) model for
  text generation using the trained encoder. Results suggest that by
  incorporating touch, the TVL model improves (+29% classification accuracy)
  touch-vision-language alignment over existing models trained on any pair of
  those modalities. Although only a small fraction of the dataset is
  human-labeled, the TVL model demonstrates improved visual-tactile understanding
  over GPT-4V (+12%) and open-source vision-language models (+32%) on a new
  touch-vision understanding benchmark. Code and data:
  https://tactile-vlm.github.io.
paper_ai_keywords: multimodal generative language model tactile data natural language labels tactile readings vision-touch pairs English language labels textual pseudo-labels GPT-4V vision-language-aligned tactile encoder open-vocabulary classification touch-vision-language (TVL) model text generation classification accuracy touch-vision-language alignment touch-vision understanding benchmark visual-tactile understanding