title: Meta-Transformer: A Unified Framework for Multimodal Learning
summary: Multimodal learning aims to build models that can process and relate
  information from multiple modalities. Despite years of development in this
  field, it still remains challenging to design a unified network for processing
  various modalities (e.g. natural language, 2D images, 3D point
  clouds, audio, video, time series, tabular data) due to the inherent gaps among
  them. In this work, we propose a framework, named Meta-Transformer, that
  leverages a frozen encoder to perform multimodal perception without
  any paired multimodal training data. In Meta-Transformer, the raw input data
  from various modalities are mapped into a shared token space, allowing a
  subsequent encoder with frozen parameters to extract high-level semantic
  features of the input data. Composed of three main components: a unified data
  tokenizer, a modality-shared encoder, and task-specific heads for downstream
  tasks, Meta-Transformer is the first framework to perform unified learning
  across 12 modalities with unpaired data. Experiments on different benchmarks
  reveal that Meta-Transformer can handle a wide range of tasks including
  fundamental perception (text, image, point cloud, audio, video), practical
  application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph,
  tabular, and time-series). Meta-Transformer indicates a promising future for
  developing unified multimodal intelligence with transformers. Code will be
  available at https://github.com/invictus717/MetaTransformer
paper_authors._id: 64ba0b16b7af2cf5151abc21
paper_authors.name: Yiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:30:12.378Z
paper_authors._id: 63176933b58b0184630d2c74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
paper_authors.fullname: Yiyuan Zhang
paper_authors.type: user
paper_authors.user: Yiyuan
paper_authors._id: 64ba0b16b7af2cf5151abc22
paper_authors.name: Kaixiong Gong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-21T07:20:24.138Z
paper_authors._id: 642e427f6748dd4f8eeb2f38
paper_authors.avatarUrl: /avatars/07158ff6aa1803c846403594c5d55a34.svg
paper_authors.fullname: Kaixiong Gong
paper_authors.type: user
paper_authors.user: kxgong
paper_authors._id: 64ba0b16b7af2cf5151abc23
paper_authors.name: Kaipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:34:59.993Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 64ba0b16b7af2cf5151abc24
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:43.920Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 64ba0b16b7af2cf5151abc25
paper_authors.name: Yu Qiao
paper_authors._id: 64ba0b16b7af2cf5151abc26
paper_authors.name: Wanli Ouyang
paper_authors._id: 64ba0b16b7af2cf5151abc27
paper_authors.name: Xiangyu Yue
paper_title: Meta-Transformer: A Unified Framework for Multimodal Learning
paper_summary: Multimodal learning aims to build models that can process and relate
  information from multiple modalities. Despite years of development in this
  field, it still remains challenging to design a unified network for processing
  various modalities (e.g. natural language, 2D images, 3D point
  clouds, audio, video, time series, tabular data) due to the inherent gaps among
  them. In this work, we propose a framework, named Meta-Transformer, that
  leverages a frozen encoder to perform multimodal perception without
  any paired multimodal training data. In Meta-Transformer, the raw input data
  from various modalities are mapped into a shared token space, allowing a
  subsequent encoder with frozen parameters to extract high-level semantic
  features of the input data. Composed of three main components: a unified data
  tokenizer, a modality-shared encoder, and task-specific heads for downstream
  tasks, Meta-Transformer is the first framework to perform unified learning
  across 12 modalities with unpaired data. Experiments on different benchmarks
  reveal that Meta-Transformer can handle a wide range of tasks including
  fundamental perception (text, image, point cloud, audio, video), practical
  application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph,
  tabular, and time-series). Meta-Transformer indicates a promising future for
  developing unified multimodal intelligence with transformers. Code will be
  available at https://github.com/invictus717/MetaTransformer
paper_ai_keywords: Meta-Transformer frozen encoder unified data tokenizer modality-shared encoder task-specific heads multimodal perception shared token space high-level semantic features