title: Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex
    Capabilities
summary: GPT-4o, an all-encompassing model, represents a milestone in the development
  of large multi-modal language models. It can understand visual, auditory, and
  textual modalities, directly output audio, and support flexible duplex
  interaction. Models from the open-source community often achieve some
  functionalities of GPT-4o, such as visual understanding and voice chat.
  Nevertheless, training a unified model that incorporates all modalities is
  challenging due to the complexities of multi-modal data, intricate model
  architectures, and training processes. In this paper, we introduce Mini-Omni2,
  a visual-audio assistant capable of providing real-time, end-to-end voice
  responses to visoin and audio queries. By integrating pretrained visual and
  auditory encoders, Mini-Omni2 maintains performance in individual modalities.
  We propose a three-stage training process to align modalities, allowing the
  language model to handle multi-modal inputs and outputs after training on a
  limited dataset. For interaction, we introduce a command-based interruption
  mechanism, enabling more flexible interaction with users. To the best of our
  knowledge, Mini-Omni2 is one of the closest reproductions of GPT-4o, which have
  similar form of functionality, and we hope it can offer valuable insights for
  subsequent research.
paper_authors._id: 6710b98b967512639d50a70e
paper_authors.name: Zhifei Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:28:54.691Z
paper_authors._id: 66c5d81a4061fd5907443787
paper_authors.avatarUrl: /avatars/2e107195b1ff7d06bbc6c9bd4e5620cf.svg
paper_authors.fullname: zhifei
paper_authors.type: user
paper_authors.user: filicos
paper_authors._id: 6710b98b967512639d50a70f
paper_authors.name: Changqiao Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:14:20.838Z
paper_authors._id: 66cdb20c6aee4c865fe9647e
paper_authors.avatarUrl: /avatars/d3637e8278c4b14d0f03ebfbfd3d3711.svg
paper_authors.fullname: Changqiao Wu
paper_authors.type: user
paper_authors.user: gpt-omni
paper_title: Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex
    Capabilities
paper_summary: GPT-4o, an all-encompassing model, represents a milestone in the development
  of large multi-modal language models. It can understand visual, auditory, and
  textual modalities, directly output audio, and support flexible duplex
  interaction. Models from the open-source community often achieve some
  functionalities of GPT-4o, such as visual understanding and voice chat.
  Nevertheless, training a unified model that incorporates all modalities is
  challenging due to the complexities of multi-modal data, intricate model
  architectures, and training processes. In this paper, we introduce Mini-Omni2,
  a visual-audio assistant capable of providing real-time, end-to-end voice
  responses to visoin and audio queries. By integrating pretrained visual and
  auditory encoders, Mini-Omni2 maintains performance in individual modalities.
  We propose a three-stage training process to align modalities, allowing the
  language model to handle multi-modal inputs and outputs after training on a
  limited dataset. For interaction, we introduce a command-based interruption
  mechanism, enabling more flexible interaction with users. To the best of our
  knowledge, Mini-Omni2 is one of the closest reproductions of GPT-4o, which have
  similar form of functionality, and we hope it can offer valuable insights for
  subsequent research.
paper_ai_summary: Mini-Omni2 is a real-time, end-to-end voice response model that integrates visual and auditory encoders and employs a three-stage training process to handle multi-modal data.
paper_ai_keywords: multi-modal language models visual-audio assistant pretrained visual encoders pretrained auditory encoders three-stage training process command-based interruption mechanism