Stream-Omni: Simultaneous Multimodal Interactions with Large
  Language-Vision-Speech Model
The emergence of GPT-4o-like large multimodal models (LMMs) has raised the
exploration of integrating text, vision, and speech modalities to support more
flexible multimodal interaction. Existing LMMs typically concatenate
representation of modalities along the sequence dimension and feed them into a
large language model (LLM) backbone. While sequence-dimension concatenation is
straightforward for modality integration, it often relies heavily on
large-scale data to learn modality alignments. In this paper, we aim to model
the relationships between modalities more purposefully, thereby achieving more
efficient and flexible modality alignments. To this end, we propose
Stream-Omni, a large language-vision-speech model with efficient modality
alignments, which can simultaneously support interactions under various
modality combinations. Stream-Omni employs LLM as the backbone and aligns the
vision and speech to the text based on their relationships. For vision that is
semantically complementary to text, Stream-Omni uses sequence-dimension
concatenation to achieve vision-text alignment. For speech that is semantically
consistent with text, Stream-Omni introduces a CTC-based layer-dimension
mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve
modality alignments with less data (especially speech), enabling the transfer
of text capabilities to other modalities. Experiments on various benchmarks
demonstrate that Stream-Omni achieves strong performance on visual
understanding, speech interaction, and vision-grounded speech interaction
tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously
provide intermediate text outputs (such as ASR transcriptions and model
responses) during speech interaction, offering users a comprehensive multimodal
experience.
685129448a68fee7f6ba4c04
Shaolei Zhang
admin_assigned
2025-06-19T15:22:54.619Z
64803e5dc57f629056c601f1
/avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
Shaolei Zhang
user
zhangshaolei
685129448a68fee7f6ba4c05
Shoutao Guo
685129448a68fee7f6ba4c06
Qingkai Fang
685129448a68fee7f6ba4c07
Yan Zhou
685129448a68fee7f6ba4c08
Yang Feng
Stream-Omni: Simultaneous Multimodal Interactions with Large
  Language-Vision-Speech Model
The emergence of GPT-4o-like large multimodal models (LMMs) has raised the
exploration of integrating text, vision, and speech modalities to support more
flexible multimodal interaction. Existing LMMs typically concatenate
representation of modalities along the sequence dimension and feed them into a
large language model (LLM) backbone. While sequence-dimension concatenation is
straightforward for modality integration, it often relies heavily on
large-scale data to learn modality alignments. In this paper, we aim to model
the relationships between modalities more purposefully, thereby achieving more
efficient and flexible modality alignments. To this end, we propose
Stream-Omni, a large language-vision-speech model with efficient modality
alignments, which can simultaneously support interactions under various
modality combinations. Stream-Omni employs LLM as the backbone and aligns the
vision and speech to the text based on their relationships. For vision that is
semantically complementary to text, Stream-Omni uses sequence-dimension
concatenation to achieve vision-text alignment. For speech that is semantically
consistent with text, Stream-Omni introduces a CTC-based layer-dimension
mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve
modality alignments with less data (especially speech), enabling the transfer
of text capabilities to other modalities. Experiments on various benchmarks
demonstrate that Stream-Omni achieves strong performance on visual
understanding, speech interaction, and vision-grounded speech interaction
tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously
provide intermediate text outputs (such as ASR transcriptions and model
responses) during speech interaction, offering users a comprehensive multimodal
experience.
Stream-Omni, a large multimodal model, integrates text, vision, and speech by efficiently aligning modalities using sequence-dimension concatenation for vision and layer-dimension mapping for speech, achieving strong performance with less data.
GPT-4o-like large multimodal models LLM backbone modality alignments sequence-dimension concatenation CTC-based layer-dimension mapping visual understanding speech interaction vision-grounded speech interaction ASR transcriptions model responses