title: TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via
    Autoregressive Diffusion Models
summary: In this paper, we present TalkingMachines -- an efficient framework that
  transforms pretrained video generation models into real-time, audio-driven
  character animators. TalkingMachines enables natural conversational experiences
  by integrating an audio large language model (LLM) with our video generation
  foundation model. Our primary contributions include: (1) We adapt a pretrained
  SOTA image-to-video DiT into an audio-driven avatar generation model of 18
  billion parameters; (2) We enable infinite video streaming without error
  accumulation through asymmetric knowledge distillation from a bidirectional
  teacher model into a sparse causal, autoregressive student model; (3) We design
  a high-throughput, low-latency inference pipeline incorporating several key
  engineering optimizations such as: (a) disaggregation of the DiT and VAE
  decoder across separate devices, (b) efficient overlap of inter-device
  communication and computation using CUDA streams, (c) elimination of redundant
  recomputations to maximize frame-generation throughput. Please see demo videos
  here - https://aaxwaz.github.io/TalkingMachines/
paper_authors._id: 684107c6142b5c0b4226025f
paper_authors.name: Chetwin Low
paper_authors._id: 684107c6142b5c0b42260260
paper_authors.name: Weimin Wang
paper_title: TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via
    Autoregressive Diffusion Models
paper_summary: In this paper, we present TalkingMachines -- an efficient framework that
  transforms pretrained video generation models into real-time, audio-driven
  character animators. TalkingMachines enables natural conversational experiences
  by integrating an audio large language model (LLM) with our video generation
  foundation model. Our primary contributions include: (1) We adapt a pretrained
  SOTA image-to-video DiT into an audio-driven avatar generation model of 18
  billion parameters; (2) We enable infinite video streaming without error
  accumulation through asymmetric knowledge distillation from a bidirectional
  teacher model into a sparse causal, autoregressive student model; (3) We design
  a high-throughput, low-latency inference pipeline incorporating several key
  engineering optimizations such as: (a) disaggregation of the DiT and VAE
  decoder across separate devices, (b) efficient overlap of inter-device
  communication and computation using CUDA streams, (c) elimination of redundant
  recomputations to maximize frame-generation throughput. Please see demo videos
  here - https://aaxwaz.github.io/TalkingMachines/
paper_ai_summary: TalkingMachines transforms a pretrained image-to-video model into an audio-driven avatar generator, supports infinite video streaming, and uses engineering optimizations for real-time performance.
paper_ai_keywords: DiT audio large language model asymmetric knowledge distillation bidirectional teacher model sparse causal autoregressive student model inference pipeline CUDA streams frame-generation throughput