title: OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
summary: Recently, human-computer interaction with various modalities has shown
  promising applications, like GPT-4o and Gemini. Given the foundational role of
  multimodal joint representation in understanding and generation pipelines,
  high-quality omni joint representations would be a step toward co-processing
  more diverse multimodal information. In this work, we present OmniBind,
  large-scale multimodal joint representation models ranging in scale from 7
  billion to 30 billion parameters, which support 3D, audio, image, and language
  inputs. Due to the scarcity of data pairs across all modalities, instead of
  training large models from scratch, we propose remapping and binding the spaces
  of various pre-trained specialist models together. This approach enables
  "scaling up" by indirectly increasing the model parameters and the amount of
  seen data. To effectively integrate various spaces, we dynamically assign
  weights to different spaces by learning routers with two objectives:
  cross-modal overall alignment and language representation decoupling. Notably,
  since binding and routing spaces both only require lightweight networks,
  OmniBind is extremely training-efficient. Learning the largest 30B model
  requires merely unpaired unimodal data and approximately 3 days on a single
  8-4090 node. Extensive experiments demonstrate the versatility and superiority
  of OmniBind as an omni representation model, highlighting its great potential
  for diverse applications, such as any-query and composable multimodal
  understanding.
paper_authors._id: 6697584b6fda02619409772d
paper_authors.name: Zehan Wang
paper_authors._id: 6697584b6fda02619409772e
paper_authors.name: Ziang Zhang
paper_authors._id: 6697584b6fda02619409772f
paper_authors.name: Hang Zhang
paper_authors._id: 6697584b6fda026194097730
paper_authors.name: Luping Liu
paper_authors._id: 6697584b6fda026194097731
paper_authors.name: Rongjie Huang
paper_authors._id: 6697584b6fda026194097732
paper_authors.name: Xize Cheng
paper_authors._id: 6697584b6fda026194097733
paper_authors.name: Hengshuang Zhao
paper_authors._id: 6697584b6fda026194097734
paper_authors.name: Zhou Zhao
paper_title: OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
paper_summary: Recently, human-computer interaction with various modalities has shown
  promising applications, like GPT-4o and Gemini. Given the foundational role of
  multimodal joint representation in understanding and generation pipelines,
  high-quality omni joint representations would be a step toward co-processing
  more diverse multimodal information. In this work, we present OmniBind,
  large-scale multimodal joint representation models ranging in scale from 7
  billion to 30 billion parameters, which support 3D, audio, image, and language
  inputs. Due to the scarcity of data pairs across all modalities, instead of
  training large models from scratch, we propose remapping and binding the spaces
  of various pre-trained specialist models together. This approach enables
  "scaling up" by indirectly increasing the model parameters and the amount of
  seen data. To effectively integrate various spaces, we dynamically assign
  weights to different spaces by learning routers with two objectives:
  cross-modal overall alignment and language representation decoupling. Notably,
  since binding and routing spaces both only require lightweight networks,
  OmniBind is extremely training-efficient. Learning the largest 30B model
  requires merely unpaired unimodal data and approximately 3 days on a single
  8-4090 node. Extensive experiments demonstrate the versatility and superiority
  of OmniBind as an omni representation model, highlighting its great potential
  for diverse applications, such as any-query and composable multimodal
  understanding.
paper_ai_summary: OmniBind integrates multimodal representations by remapping and binding pre-trained models, achieving efficient scaling and superior performance across various tasks.
paper_ai_keywords: multimodal joint representation OmniBind pre-trained specialist models cross-modal overall alignment language representation decoupling routing networks training efficiency any-query understanding composable multimodal understanding