title: V-Express: Conditional Dropout for Progressive Training of Portrait
    Video Generation
summary: In the field of portrait video generation, the use of single images to
  generate portrait videos has become increasingly prevalent. A common approach
  involves leveraging generative models to enhance adapters for controlled
  generation. However, control signals (e.g., text, audio, reference image, pose,
  depth map, etc.) can vary in strength. Among these, weaker conditions often
  struggle to be effective due to interference from stronger conditions, posing a
  challenge in balancing these conditions. In our work on portrait video
  generation, we identified audio signals as particularly weak, often
  overshadowed by stronger signals such as facial pose and reference image.
  However, direct training with weak signals often leads to difficulties in
  convergence. To address this, we propose V-Express, a simple method that
  balances different control signals through the progressive training and the
  conditional dropout operation. Our method gradually enables effective control
  by weak conditions, thereby achieving generation capabilities that
  simultaneously take into account the facial pose, reference image, and audio.
  The experimental results demonstrate that our method can effectively generate
  portrait videos controlled by audio. Furthermore, a potential solution is
  provided for the simultaneous and effective use of conditions of varying
  strengths.
paper_authors._id: 665fe5fe3be5defb09f3d6fc
paper_authors.name: Cong Wang
paper_authors._id: 665fe5fe3be5defb09f3d6fd
paper_authors.name: Kuan Tian
paper_authors._id: 665fe5fe3be5defb09f3d6fe
paper_authors.name: Jun Zhang
paper_authors._id: 665fe5fe3be5defb09f3d6ff
paper_authors.name: Yonghang Guan
paper_authors._id: 665fe5fe3be5defb09f3d700
paper_authors.name: Feng Luo
paper_authors._id: 665fe5fe3be5defb09f3d701
paper_authors.name: Fei Shen
paper_authors._id: 665fe5fe3be5defb09f3d702
paper_authors.name: Zhiwei Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T10:43:26.914Z
paper_authors._id: 66446066b5a21591719f3f8e
paper_authors.avatarUrl: /avatars/923e7786da7bdaa69c7e3d1567a6a156.svg
paper_authors.fullname: Jiang Zhiwei
paper_authors.type: user
paper_authors.user: Laziobird
paper_authors._id: 665fe5fe3be5defb09f3d703
paper_authors.name: Qing Gu
paper_authors._id: 665fe5fe3be5defb09f3d704
paper_authors.name: Xiao Han
paper_authors._id: 665fe5fe3be5defb09f3d705
paper_authors.name: Wei Yang
paper_title: V-Express: Conditional Dropout for Progressive Training of Portrait
    Video Generation
paper_summary: In the field of portrait video generation, the use of single images to
  generate portrait videos has become increasingly prevalent. A common approach
  involves leveraging generative models to enhance adapters for controlled
  generation. However, control signals (e.g., text, audio, reference image, pose,
  depth map, etc.) can vary in strength. Among these, weaker conditions often
  struggle to be effective due to interference from stronger conditions, posing a
  challenge in balancing these conditions. In our work on portrait video
  generation, we identified audio signals as particularly weak, often
  overshadowed by stronger signals such as facial pose and reference image.
  However, direct training with weak signals often leads to difficulties in
  convergence. To address this, we propose V-Express, a simple method that
  balances different control signals through the progressive training and the
  conditional dropout operation. Our method gradually enables effective control
  by weak conditions, thereby achieving generation capabilities that
  simultaneously take into account the facial pose, reference image, and audio.
  The experimental results demonstrate that our method can effectively generate
  portrait videos controlled by audio. Furthermore, a potential solution is
  provided for the simultaneous and effective use of conditions of varying
  strengths.
paper_ai_summary: V-Express addresses the challenge of weak control signals in portrait video generation through progressive training and conditional dropout, enabling effective audio-controlled video generation alongside other conditions.
paper_ai_keywords: V-Express progressive training conditional dropout portrait video generation control signals audio signals facial pose reference image