title: Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant
summary: Large Language Models (LLMs) have revolutionized natural language processing,
  but their application to speech-based tasks remains challenging due to the
  complexities of integrating audio and text modalities. This paper introduces
  Ichigo, a mixed-modal model that seamlessly processes interleaved sequences of
  speech and text. Utilizing a tokenized early-fusion approach, Ichigo quantizes
  speech into discrete tokens and employs a uniform transformer-based
  architecture for both speech and text modalities. This method enables joint
  reasoning and generation across modalities without the need for separate
  adapters. We present a comprehensive training methodology, including
  pre-training on multilingual speech recognition datasets and fine-tuning on a
  curated instruction dataset. Ichigo demonstrates state-of-the-art performance
  on speech question-answering benchmarks, outperforming existing open-source
  speech language models and achieving comparable results to cascaded systems.
  Notably, Ichigo exhibits a latency of just 111 ms to first token generation,
  significantly lower than current models. Our approach not only advances the
  field of multimodal AI but also provides a framework for smaller research teams
  to contribute effectively to open-source speech-language models.
paper_authors._id: 67178d2465aee70effea04ae
paper_authors.name: Alan Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:29:03.628Z
paper_authors._id: 62d7b2339b629105a5d6888a
paper_authors.avatarUrl: /avatars/c3f164fde6b8f9a671890e08ce8a3e75.svg
paper_authors.fullname: Alan Dao
paper_authors.type: user
paper_authors.user: alandao
paper_authors._id: 67178d2465aee70effea04af
paper_authors.name: Dinh Bach Vu
paper_authors._id: 67178d2465aee70effea04b0
paper_authors.name: Huy Hoang Ha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T12:43:59.290Z
paper_authors._id: 630a5ef0e81e1dea2cedcec0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630a5ef0e81e1dea2cedcec0/ATtyCvYoX4z7uxsm2sJU2.png
paper_authors.fullname: Hà Huy Hoàng
paper_authors.type: user
paper_authors.user: HoangHa
paper_title: Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant
paper_summary: Large Language Models (LLMs) have revolutionized natural language processing,
  but their application to speech-based tasks remains challenging due to the
  complexities of integrating audio and text modalities. This paper introduces
  Ichigo, a mixed-modal model that seamlessly processes interleaved sequences of
  speech and text. Utilizing a tokenized early-fusion approach, Ichigo quantizes
  speech into discrete tokens and employs a uniform transformer-based
  architecture for both speech and text modalities. This method enables joint
  reasoning and generation across modalities without the need for separate
  adapters. We present a comprehensive training methodology, including
  pre-training on multilingual speech recognition datasets and fine-tuning on a
  curated instruction dataset. Ichigo demonstrates state-of-the-art performance
  on speech question-answering benchmarks, outperforming existing open-source
  speech language models and achieving comparable results to cascaded systems.
  Notably, Ichigo exhibits a latency of just 111 ms to first token generation,
  significantly lower than current models. Our approach not only advances the
  field of multimodal AI but also provides a framework for smaller research teams
  to contribute effectively to open-source speech-language models.
paper_ai_keywords: mixed-modal model tokenized early-fusion approach uniform transformer-based architecture joint reasoning generation across modalities pre-training multilingual speech recognition datasets fine-tuning speech question-answering benchmarks cascaded systems latency first token generation open-source speech language models