title: Language Model Can Listen While Speaking
summary: Dialogue serves as the most natural manner of human-computer interaction
  (HCI). Recent advancements in speech language models (SLM) have significantly
  enhanced speech-based conversational AI. However, these models are limited to
  turn-based conversation, lacking the ability to interact with humans in
  real-time spoken scenarios, for example, being interrupted when the generated
  content is not satisfactory. To address these limitations, we explore full
  duplex modeling (FDM) in interactive speech language models (iSLM), focusing on
  enhancing real-time interaction and, more explicitly, exploring the
  quintessential ability of interruption. We introduce a novel model design,
  namely listening-while-speaking language model (LSLM), an end-to-end system
  equipped with both listening and speaking channels. Our LSLM employs a
  token-based decoder-only TTS for speech generation and a streaming
  self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses
  both channels for autoregressive generation and detects turn-taking in real
  time. Three fusion strategies -- early fusion, middle fusion, and late fusion
  -- are explored, with middle fusion achieving an optimal balance between speech
  generation and real-time interaction. Two experimental settings, command-based
  FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity
  to diverse instructions. Our results highlight LSLM's capability to achieve
  duplex communication with minimal impact on existing systems. This study aims
  to advance the development of interactive speech dialogue systems, enhancing
  their applicability in real-world contexts.
paper_authors._id: 66b197769aedfcdb8f5c19f9
paper_authors.name: Ziyang Ma
paper_authors._id: 66b197769aedfcdb8f5c19fa
paper_authors.name: Yakun Song
paper_authors._id: 66b197769aedfcdb8f5c19fb
paper_authors.name: Chenpeng Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:38:48.163Z
paper_authors._id: 661ae5f7040d38d24bdd48e0
paper_authors.avatarUrl: /avatars/79a0cc8c5bae3c422160002dbc2869ce.svg
paper_authors.fullname: Chenpeng Du
paper_authors.type: user
paper_authors.user: cpdu
paper_authors._id: 66b197769aedfcdb8f5c19fc
paper_authors.name: Jian Cong
paper_authors._id: 66b197769aedfcdb8f5c19fd
paper_authors.name: Zhuo Chen
paper_authors._id: 66b197769aedfcdb8f5c19fe
paper_authors.name: Yuping Wang
paper_authors._id: 66b197769aedfcdb8f5c19ff
paper_authors.name: Yuxuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T13:18:23.971Z
paper_authors._id: 6694e5344a34b68738070c07
paper_authors.avatarUrl: /avatars/f33ec08ec9822bb353020000a2fcbd84.svg
paper_authors.fullname: Yuxuan WANG
paper_authors.type: user
paper_authors.user: AlbusYuxuanWang
paper_authors._id: 66b197769aedfcdb8f5c1a00
paper_authors.name: Xie Chen
paper_title: Language Model Can Listen While Speaking
paper_summary: Dialogue serves as the most natural manner of human-computer interaction
  (HCI). Recent advancements in speech language models (SLM) have significantly
  enhanced speech-based conversational AI. However, these models are limited to
  turn-based conversation, lacking the ability to interact with humans in
  real-time spoken scenarios, for example, being interrupted when the generated
  content is not satisfactory. To address these limitations, we explore full
  duplex modeling (FDM) in interactive speech language models (iSLM), focusing on
  enhancing real-time interaction and, more explicitly, exploring the
  quintessential ability of interruption. We introduce a novel model design,
  namely listening-while-speaking language model (LSLM), an end-to-end system
  equipped with both listening and speaking channels. Our LSLM employs a
  token-based decoder-only TTS for speech generation and a streaming
  self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses
  both channels for autoregressive generation and detects turn-taking in real
  time. Three fusion strategies -- early fusion, middle fusion, and late fusion
  -- are explored, with middle fusion achieving an optimal balance between speech
  generation and real-time interaction. Two experimental settings, command-based
  FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity
  to diverse instructions. Our results highlight LSLM's capability to achieve
  duplex communication with minimal impact on existing systems. This study aims
  to advance the development of interactive speech dialogue systems, enhancing
  their applicability in real-world contexts.
paper_ai_summary: A novel listening-while-speaking language model (LSLM) enhances real-time, full-duplex speech interaction by integrating speech generation and real-time audio input with an optimal fusion strategy.
paper_ai_keywords: full duplex modeling interactive speech language models listening-while-speaking language model token-based decoder-only TTS streaming self-supervised learning early fusion middle fusion late fusion duplex communication