title: Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive
    Modality Alignment
summary: Recent advances in large language models, particularly following GPT-4o, have
  sparked increasing interest in developing omni-modal models capable of
  understanding more modalities. While some open-source alternatives have
  emerged, there is still a notable lag behind specialized single-modality models
  in performance. In this paper, we present Ola, an Omni-modal language model
  that achieves competitive performance across image, video, and audio
  understanding compared to specialized counterparts. The core design of Ola lies
  in its progressive modality alignment strategy that extends the supporting
  modality of the language model progressively. Our training pipeline begins with
  the most distinct modalities: image and text, then gradually expands the skill
  sets of the model using speech data that connects language and audio knowledge,
  and video data that connects all modalities. The progressive learning pipeline
  also enables us to maintain a relatively small size of the cross-modal
  alignment data, making developing omni-modal from existing vision-language
  models easy and less costly. Moreover, to unlock an advanced interactive
  experience like GPT-4o, we further design a sentence-wise decoding solution for
  streaming speech generation. Extensive experiments demonstrate that Ola
  surpasses existing open omni-modal LLMs across all modalities while achieving
  highly competitive performance compared to state-of-the-art specialized models
  of similar sizes. We aim to make Ola a fully open omni-modal understanding
  solution to advance future research in this emerging field. Model weights,
  code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
paper_authors._id: 67a586fad177de2eeba7de7b
paper_authors.name: Zuyan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:58:10.679Z
paper_authors._id: 64f001bfabd9fb1914398bd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
paper_authors.fullname: liuzuyan
paper_authors.type: user
paper_authors.user: Zuyan
paper_authors._id: 67a586fad177de2eeba7de7c
paper_authors.name: Yuhao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:37:45.556Z
paper_authors._id: 652965773a416e1f2173443b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
paper_authors.fullname: Yuhao Dong
paper_authors.type: user
paper_authors.user: THUdyh
paper_authors._id: 67a586fad177de2eeba7de7d
paper_authors.name: Jiahui Wang
paper_authors._id: 67a586fad177de2eeba7de7e
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:38:14.852Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 67a586fad177de2eeba7de7f
paper_authors.name: Winston Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:38:23.468Z
paper_authors._id: 63673bb9d0ee6e2662be0ec1
paper_authors.avatarUrl: /avatars/1b8976785d64bc4e3f7159ccdb7f06c5.svg
paper_authors.fullname: Qingqiao Hu
paper_authors.type: user
paper_authors.user: WinstonHu
paper_authors._id: 67a586fad177de2eeba7de80
paper_authors.name: Jiwen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:38:29.456Z
paper_authors._id: 66c44203ea476bea05e9fcd7
paper_authors.avatarUrl: /avatars/b061eebec609446e669f5ad6365959f9.svg
paper_authors.fullname: lu
paper_authors.type: user
paper_authors.user: jiwenlu
paper_authors._id: 67a586fad177de2eeba7de81
paper_authors.name: Yongming Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:38:35.766Z
paper_authors._id: 63e4865354f51ea342d45d78
paper_authors.avatarUrl: /avatars/2e7eccc878751331ca8b282f53e38899.svg
paper_authors.fullname: Yongming Rao
paper_authors.type: user
paper_authors.user: raoyongming
paper_title: Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive
    Modality Alignment
paper_summary: Recent advances in large language models, particularly following GPT-4o, have
  sparked increasing interest in developing omni-modal models capable of
  understanding more modalities. While some open-source alternatives have
  emerged, there is still a notable lag behind specialized single-modality models
  in performance. In this paper, we present Ola, an Omni-modal language model
  that achieves competitive performance across image, video, and audio
  understanding compared to specialized counterparts. The core design of Ola lies
  in its progressive modality alignment strategy that extends the supporting
  modality of the language model progressively. Our training pipeline begins with
  the most distinct modalities: image and text, then gradually expands the skill
  sets of the model using speech data that connects language and audio knowledge,
  and video data that connects all modalities. The progressive learning pipeline
  also enables us to maintain a relatively small size of the cross-modal
  alignment data, making developing omni-modal from existing vision-language
  models easy and less costly. Moreover, to unlock an advanced interactive
  experience like GPT-4o, we further design a sentence-wise decoding solution for
  streaming speech generation. Extensive experiments demonstrate that Ola
  surpasses existing open omni-modal LLMs across all modalities while achieving
  highly competitive performance compared to state-of-the-art specialized models
  of similar sizes. We aim to make Ola a fully open omni-modal understanding
  solution to advance future research in this emerging field. Model weights,
  code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
paper_ai_summary: Ola is an omni-modal language model that achieves competitive performance across image, video, and audio understanding through progressive modality alignment and efficient cross-modal training.
paper_ai_keywords: omni-modal language model progressive modality alignment cross-modal alignment vision-language models streaming speech generation