title: Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex
    Models
summary: As large language models (LLMs) increasingly permeate daily lives, there is a
  growing demand for real-time interactions that mirror human conversations.
  Traditional turn-based chat systems driven by LLMs prevent users from verbally
  interacting with the system while it is generating responses. To overcome these
  limitations, we adapt existing LLMs to duplex models so that these
  LLMs can listen for users while generating output and dynamically adjust
  themselves to provide users with instant feedback. % such as in response to
  interruptions. Specifically, we divide the queries and responses of
  conversations into several time slices and then adopt a
  time-division-multiplexing (TDM) encoding-decoding strategy to
  pseudo-simultaneously process these slices. Furthermore, to make LLMs
  proficient enough to handle real-time conversations, we build a fine-tuning
  dataset consisting of alternating time slices of queries and responses as well
  as covering typical feedback types in instantaneous interactions. Our
  experiments show that although the queries and responses of conversations are
  segmented into incomplete slices for processing, LLMs can preserve their
  original performance on standard benchmarks with a few fine-tuning steps on our
  dataset. Automatic and human evaluation indicate that duplex models make
  user-AI interactions more natural and human-like, and greatly improve user
  satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
  released.
paper_authors._id: 667a90b03bf5064fe3367191
paper_authors.name: Xinrong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:59:38.360Z
paper_authors._id: 6316eec27b0ee0136e5e8132
paper_authors.avatarUrl: /avatars/a08f7cbaf155d8cef3dd6197a21890a5.svg
paper_authors.fullname: xinrongzhang
paper_authors.type: user
paper_authors.user: xinrongzhang2022
paper_authors._id: 667a90b03bf5064fe3367192
paper_authors.name: Yingfa Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:59:44.450Z
paper_authors._id: 6144e4667f2544bb450787b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
paper_authors.fullname: Yingfa Chen
paper_authors.type: user
paper_authors.user: chen-yingfa
paper_authors._id: 667a90b03bf5064fe3367193
paper_authors.name: Shengding Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:59:50.264Z
paper_authors._id: 6041bec6c15e823a685f2afc
paper_authors.avatarUrl: /avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
paper_authors.fullname: Shengding Hu
paper_authors.type: user
paper_authors.user: ShengdingHu
paper_authors._id: 667a90b03bf5064fe3367194
paper_authors.name: Xu Han
paper_authors._id: 667a90b03bf5064fe3367195
paper_authors.name: Zihang Xu
paper_authors._id: 667a90b03bf5064fe3367196
paper_authors.name: Yuanwei Xu
paper_authors._id: 667a90b03bf5064fe3367197
paper_authors.name: Weilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T14:00:34.382Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 667a90b03bf5064fe3367198
paper_authors.name: Maosong Sun
paper_authors._id: 667a90b03bf5064fe3367199
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T14:01:19.677Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_title: Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex
    Models
paper_summary: As large language models (LLMs) increasingly permeate daily lives, there is a
  growing demand for real-time interactions that mirror human conversations.
  Traditional turn-based chat systems driven by LLMs prevent users from verbally
  interacting with the system while it is generating responses. To overcome these
  limitations, we adapt existing LLMs to duplex models so that these
  LLMs can listen for users while generating output and dynamically adjust
  themselves to provide users with instant feedback. % such as in response to
  interruptions. Specifically, we divide the queries and responses of
  conversations into several time slices and then adopt a
  time-division-multiplexing (TDM) encoding-decoding strategy to
  pseudo-simultaneously process these slices. Furthermore, to make LLMs
  proficient enough to handle real-time conversations, we build a fine-tuning
  dataset consisting of alternating time slices of queries and responses as well
  as covering typical feedback types in instantaneous interactions. Our
  experiments show that although the queries and responses of conversations are
  segmented into incomplete slices for processing, LLMs can preserve their
  original performance on standard benchmarks with a few fine-tuning steps on our
  dataset. Automatic and human evaluation indicate that duplex models make
  user-AI interactions more natural and human-like, and greatly improve user
  satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
  released.
paper_ai_summary: Duplex models enable real-time, simultaneous interactions for LLMs through time-division-multiplexing, improving user satisfaction and naturalness in AI conversations.
paper_ai_keywords: LLMs duplex models time-division-multiplexing (TDM) encoding-decoding strategy time slices fine-tuning standard benchmarks user-AI interactions