title: DiTaiListener: Controllable High Fidelity Listener Video Generation with
    Diffusion
summary: Generating naturalistic and nuanced listener motions for extended
  interactions remains an open problem. Existing methods often rely on
  low-dimensional motion codes for facial behavior generation followed by
  photorealistic rendering, limiting both visual fidelity and expressive
  richness. To address these challenges, we introduce DiTaiListener, powered by a
  video diffusion model with multimodal conditions. Our approach first generates
  short segments of listener responses conditioned on the speaker's speech and
  facial motions with DiTaiListener-Gen. It then refines the transitional frames
  via DiTaiListener-Edit for a seamless transition. Specifically,
  DiTaiListener-Gen adapts a Diffusion Transformer (DiT) for the task of listener
  head portrait generation by introducing a Causal Temporal Multimodal Adapter
  (CTM-Adapter) to process speakers' auditory and visual cues. CTM-Adapter
  integrates speakers' input in a causal manner into the video generation process
  to ensure temporally coherent listener responses. For long-form video
  generation, we introduce DiTaiListener-Edit, a transition refinement
  video-to-video diffusion model. The model fuses video segments into smooth and
  continuous videos, ensuring temporal consistency in facial expressions and
  image quality when merging short video segments produced by DiTaiListener-Gen.
  Quantitatively, DiTaiListener achieves the state-of-the-art performance on
  benchmark datasets in both photorealism (+73.8% in FID on RealTalk) and motion
  representation (+6.1% in FD metric on VICO) spaces. User studies confirm the
  superior performance of DiTaiListener, with the model being the clear
  preference in terms of feedback, diversity, and smoothness, outperforming
  competitors by a significant margin.
paper_authors._id: 67f766cb1879ad2f13bee3d1
paper_authors.name: Maksim Siniukov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:10:42.914Z
paper_authors._id: 63c9f93cdfac8071d01ed56f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674180895772-noauth.jpeg
paper_authors.fullname: Maksim Siniukov
paper_authors.type: user
paper_authors.user: havent-invented
paper_authors._id: 67f766cb1879ad2f13bee3d2
paper_authors.name: Di Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:10:49.652Z
paper_authors._id: 64a5d8219f3b568c202b3137
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a5d8219f3b568c202b3137/TI20Z1lWHzZpLsMkayDbU.png
paper_authors.fullname: Di Chang
paper_authors.type: user
paper_authors.user: Boese0601
paper_authors._id: 67f766cb1879ad2f13bee3d3
paper_authors.name: Minh Tran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:10:55.537Z
paper_authors._id: 632b6c08ca316c73cd3e4d8c
paper_authors.avatarUrl: /avatars/a02aa14823dd729df0267a9b55779edd.svg
paper_authors.fullname: Minh Tran
paper_authors.type: user
paper_authors.user: minhtran
paper_authors._id: 67f766cb1879ad2f13bee3d4
paper_authors.name: Hongkun Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:11:01.857Z
paper_authors._id: 6605cfc7b85b7b4ea506a33d
paper_authors.avatarUrl: /avatars/92abda656abf2298c9d28b9b2e3643a3.svg
paper_authors.fullname: Hongkun Gong
paper_authors.type: user
paper_authors.user: hongkung
paper_authors._id: 67f766cb1879ad2f13bee3d5
paper_authors.name: Ashutosh Chaubey
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:11:08.391Z
paper_authors._id: 6541185dbd60d2bd193f7999
paper_authors.avatarUrl: /avatars/4ce4a0feff9bfbb87e9f40431718ba00.svg
paper_authors.fullname: Ashutosh Chaubey
paper_authors.type: user
paper_authors.user: chaubeyG
paper_authors._id: 67f766cb1879ad2f13bee3d6
paper_authors.name: Mohammad Soleymani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:11:15.030Z
paper_authors._id: 65fcb99d383d3f256c3a92d2
paper_authors.avatarUrl: /avatars/b85d32f4d7a19816b8d499e05b173ad1.svg
paper_authors.fullname: Mohammad Soleymani
paper_authors.type: user
paper_authors.user: msoleymani
paper_title: DiTaiListener: Controllable High Fidelity Listener Video Generation with
    Diffusion
paper_summary: Generating naturalistic and nuanced listener motions for extended
  interactions remains an open problem. Existing methods often rely on
  low-dimensional motion codes for facial behavior generation followed by
  photorealistic rendering, limiting both visual fidelity and expressive
  richness. To address these challenges, we introduce DiTaiListener, powered by a
  video diffusion model with multimodal conditions. Our approach first generates
  short segments of listener responses conditioned on the speaker's speech and
  facial motions with DiTaiListener-Gen. It then refines the transitional frames
  via DiTaiListener-Edit for a seamless transition. Specifically,
  DiTaiListener-Gen adapts a Diffusion Transformer (DiT) for the task of listener
  head portrait generation by introducing a Causal Temporal Multimodal Adapter
  (CTM-Adapter) to process speakers' auditory and visual cues. CTM-Adapter
  integrates speakers' input in a causal manner into the video generation process
  to ensure temporally coherent listener responses. For long-form video
  generation, we introduce DiTaiListener-Edit, a transition refinement
  video-to-video diffusion model. The model fuses video segments into smooth and
  continuous videos, ensuring temporal consistency in facial expressions and
  image quality when merging short video segments produced by DiTaiListener-Gen.
  Quantitatively, DiTaiListener achieves the state-of-the-art performance on
  benchmark datasets in both photorealism (+73.8% in FID on RealTalk) and motion
  representation (+6.1% in FD metric on VICO) spaces. User studies confirm the
  superior performance of DiTaiListener, with the model being the clear
  preference in terms of feedback, diversity, and smoothness, outperforming
  competitors by a significant margin.
paper_ai_summary: DiTaiListener generates realistic listener motions using a video diffusion model with multimodal conditions, achieving superior photorealism and motion representation compared to existing methods.
paper_ai_keywords: video diffusion model multimodal conditions Diffusion Transformer (DiT) Causal Temporal Multimodal Adapter (CTM-Adapter) video-to-video diffusion model short video segments temporal consistency photorealism motion representation RealTalk VICO user studies