title: FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking
    Portrait
summary: With the rapid advancement of diffusion-based generative models, portrait
  image animation has achieved remarkable results. However, it still faces
  challenges in temporally consistent video generation and fast sampling due to
  its iterative sampling nature. This paper presents FLOAT, an audio-driven
  talking portrait video generation method based on flow matching generative
  model. We shift the generative modeling from the pixel-based latent space to a
  learned motion latent space, enabling efficient design of temporally consistent
  motion. To achieve this, we introduce a transformer-based vector field
  predictor with a simple yet effective frame-wise conditioning mechanism.
  Additionally, our method supports speech-driven emotion enhancement, enabling a
  natural incorporation of expressive motions. Extensive experiments demonstrate
  that our method outperforms state-of-the-art audio-driven talking portrait
  methods in terms of visual quality, motion fidelity, and efficiency.
paper_authors._id: 674e9ca6068c8178f4815e4c
paper_authors.name: Taekyung Ki
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:46:17.846Z
paper_authors._id: 66b57c77778c98d29446c8ec
paper_authors.avatarUrl: /avatars/63a7da38ee3808858f0f786a3a4a8dae.svg
paper_authors.fullname: Taekyung Ki
paper_authors.type: user
paper_authors.user: tkkitkki
paper_authors._id: 674e9ca6068c8178f4815e4d
paper_authors.name: Dongchan Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:24:09.885Z
paper_authors._id: 663cb73c749d810d11ab7d32
paper_authors.avatarUrl: /avatars/c95a9e07e23e16e3f362066f8478908e.svg
paper_authors.fullname: dongchan min
paper_authors.type: user
paper_authors.user: kevin0417
paper_authors._id: 674e9ca6068c8178f4815e4e
paper_authors.name: Gyoungsu Chae
paper_title: FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking
    Portrait
paper_summary: With the rapid advancement of diffusion-based generative models, portrait
  image animation has achieved remarkable results. However, it still faces
  challenges in temporally consistent video generation and fast sampling due to
  its iterative sampling nature. This paper presents FLOAT, an audio-driven
  talking portrait video generation method based on flow matching generative
  model. We shift the generative modeling from the pixel-based latent space to a
  learned motion latent space, enabling efficient design of temporally consistent
  motion. To achieve this, we introduce a transformer-based vector field
  predictor with a simple yet effective frame-wise conditioning mechanism.
  Additionally, our method supports speech-driven emotion enhancement, enabling a
  natural incorporation of expressive motions. Extensive experiments demonstrate
  that our method outperforms state-of-the-art audio-driven talking portrait
  methods in terms of visual quality, motion fidelity, and efficiency.
paper_ai_summary:  FLOAT method generates high-quality, temporally consistent, and emotion-enhanced talking portraits using flow matching in a learned motion latent space with a transformer-based vector field predictor.
paper_ai_keywords: diffusion-based generative models flow matching generative model motion latent space transformer-based vector field predictor frame-wise conditioning mechanism