OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Recently, human-computer interaction with various modalities has shown
promising applications, like GPT-4o and Gemini. Given the foundational role of
multimodal joint representation in understanding and generation pipelines,
high-quality omni joint representations would be a step toward co-processing
more diverse multimodal information. In this work, we present OmniBind,
large-scale multimodal joint representation models ranging in scale from 7
billion to 30 billion parameters, which support 3D, audio, image, and language
inputs. Due to the scarcity of data pairs across all modalities, instead of
training large models from scratch, we propose remapping and binding the spaces
of various pre-trained specialist models together. This approach enables
"scaling up" by indirectly increasing the model parameters and the amount of
seen data. To effectively integrate various spaces, we dynamically assign
weights to different spaces by learning routers with two objectives:
cross-modal overall alignment and language representation decoupling. Notably,
since binding and routing spaces both only require lightweight networks,
OmniBind is extremely training-efficient. Learning the largest 30B model
requires merely unpaired unimodal data and approximately 3 days on a single
8-4090 node. Extensive experiments demonstrate the versatility and superiority
of OmniBind as an omni representation model, highlighting its great potential
for diverse applications, such as any-query and composable multimodal
understanding.
6697584b6fda02619409772d
Zehan Wang
6697584b6fda02619409772e
Ziang Zhang
6697584b6fda02619409772f
Hang Zhang
6697584b6fda026194097730
Luping Liu
6697584b6fda026194097731
Rongjie Huang
6697584b6fda026194097732
Xize Cheng
6697584b6fda026194097733
Hengshuang Zhao
6697584b6fda026194097734
Zhou Zhao
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Recently, human-computer interaction with various modalities has shown
promising applications, like GPT-4o and Gemini. Given the foundational role of
multimodal joint representation in understanding and generation pipelines,
high-quality omni joint representations would be a step toward co-processing
more diverse multimodal information. In this work, we present OmniBind,
large-scale multimodal joint representation models ranging in scale from 7
billion to 30 billion parameters, which support 3D, audio, image, and language
inputs. Due to the scarcity of data pairs across all modalities, instead of
training large models from scratch, we propose remapping and binding the spaces
of various pre-trained specialist models together. This approach enables
"scaling up" by indirectly increasing the model parameters and the amount of
seen data. To effectively integrate various spaces, we dynamically assign
weights to different spaces by learning routers with two objectives:
cross-modal overall alignment and language representation decoupling. Notably,
since binding and routing spaces both only require lightweight networks,
OmniBind is extremely training-efficient. Learning the largest 30B model
requires merely unpaired unimodal data and approximately 3 days on a single
8-4090 node. Extensive experiments demonstrate the versatility and superiority
of OmniBind as an omni representation model, highlighting its great potential
for diverse applications, such as any-query and composable multimodal
understanding.
OmniBind integrates multimodal representations by remapping and binding pre-trained models, achieving efficient scaling and superior performance across various tasks.
multimodal joint representation OmniBind pre-trained specialist models cross-modal overall alignment language representation decoupling routing networks training efficiency any-query understanding composable multimodal understanding