title: Multimodal Pathway: Improve Transformers with Irrelevant Data from Other
    Modalities
summary: We propose to improve transformers of a specific modality with irrelevant
  data from other modalities, e.g., improve an ImageNet model with audio or point
  cloud datasets. We would like to highlight that the data samples of the target
  modality are irrelevant to the other modalities, which distinguishes our method
  from other works utilizing paired (e.g., CLIP) or interleaved data of different
  modalities. We propose a methodology named Multimodal Pathway - given a target
  modality and a transformer designed for it, we use an auxiliary transformer
  trained with data of another modality and construct pathways to connect
  components of the two models so that data of the target modality can be
  processed by both models. In this way, we utilize the universal
  sequence-to-sequence modeling abilities of transformers obtained from two
  modalities. As a concrete implementation, we use a modality-specific tokenizer
  and task-specific head as usual but utilize the transformer blocks of the
  auxiliary model via a proposed method named Cross-Modal Re-parameterization,
  which exploits the auxiliary weights without any inference costs. On the image,
  point cloud, video, and audio recognition tasks, we observe significant and
  consistent performance improvements with irrelevant data from other modalities.
  The code and models are available at https://github.com/AILab-CVC/M2PT.
paper_authors._id: 65b338f98c7cfb1b8bc524fa
paper_authors.name: Yiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:50:33.141Z
paper_authors._id: 63176933b58b0184630d2c74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
paper_authors.fullname: Yiyuan Zhang
paper_authors.type: user
paper_authors.user: Yiyuan
paper_authors._id: 65b338f98c7cfb1b8bc524fb
paper_authors.name: Xiaohan Ding
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-26T09:18:37.159Z
paper_authors._id: 6564f049957ac688429a6754
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nE6q5TlyLKlJpbxCKoDN8.png
paper_authors.fullname: xiaohanding
paper_authors.type: user
paper_authors.user: DingXiaoH
paper_authors._id: 65b338f98c7cfb1b8bc524fc
paper_authors.name: Kaixiong Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:50:40.131Z
paper_authors._id: 642e427f6748dd4f8eeb2f38
paper_authors.avatarUrl: /avatars/07158ff6aa1803c846403594c5d55a34.svg
paper_authors.fullname: Kaixiong Gong
paper_authors.type: user
paper_authors.user: kxgong
paper_authors._id: 65b338f98c7cfb1b8bc524fd
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:50:46.635Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 65b338f98c7cfb1b8bc524fe
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:51:17.615Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 65b338f98c7cfb1b8bc524ff
paper_authors.name: Xiangyu Yue
paper_title: Multimodal Pathway: Improve Transformers with Irrelevant Data from Other
    Modalities
paper_summary: We propose to improve transformers of a specific modality with irrelevant
  data from other modalities, e.g., improve an ImageNet model with audio or point
  cloud datasets. We would like to highlight that the data samples of the target
  modality are irrelevant to the other modalities, which distinguishes our method
  from other works utilizing paired (e.g., CLIP) or interleaved data of different
  modalities. We propose a methodology named Multimodal Pathway - given a target
  modality and a transformer designed for it, we use an auxiliary transformer
  trained with data of another modality and construct pathways to connect
  components of the two models so that data of the target modality can be
  processed by both models. In this way, we utilize the universal
  sequence-to-sequence modeling abilities of transformers obtained from two
  modalities. As a concrete implementation, we use a modality-specific tokenizer
  and task-specific head as usual but utilize the transformer blocks of the
  auxiliary model via a proposed method named Cross-Modal Re-parameterization,
  which exploits the auxiliary weights without any inference costs. On the image,
  point cloud, video, and audio recognition tasks, we observe significant and
  consistent performance improvements with irrelevant data from other modalities.
  The code and models are available at https://github.com/AILab-CVC/M2PT.
paper_ai_keywords: transformers modality data samples paired datasets visualization representation engineering (RepE) pathway components universal sequence-to-sequence modeling transformer blocks modality-specific task-specific head Cross-Modal Re-parameterization image recognition point cloud recognition video recognition audio recognition irrelevant data inference costs