title: Portrait Video Editing Empowered by Multimodal Generative Priors
summary: We introduce PortraitGen, a powerful portrait video editing method that
  achieves consistent and expressive stylization with multimodal prompts.
  Traditional portrait video editing methods often struggle with 3D and temporal
  consistency, and typically lack in rendering quality and efficiency. To address
  these issues, we lift the portrait video frames to a unified dynamic 3D
  Gaussian field, which ensures structural and temporal coherence across frames.
  Furthermore, we design a novel Neural Gaussian Texture mechanism that not only
  enables sophisticated style editing but also achieves rendering speed over
  100FPS. Our approach incorporates multimodal inputs through knowledge distilled
  from large-scale 2D generative models. Our system also incorporates expression
  similarity guidance and a face-aware portrait editing module, effectively
  mitigating degradation issues associated with iterative dataset updates.
  Extensive experiments demonstrate the temporal consistency, editing efficiency,
  and superior rendering quality of our method. The broad applicability of the
  proposed approach is demonstrated through various applications, including
  text-driven editing, image-driven editing, and relighting, highlighting its
  great potential to advance the field of video editing. Demo videos and released
  code are provided in our project page: https://ustc3dv.github.io/PortraitGen/
paper_authors._id: 66f0c86916bffdccd54da55b
paper_authors.name: Xuan Gao
paper_authors._id: 66f0c86916bffdccd54da55c
paper_authors.name: Haiyao Xiao
paper_authors._id: 66f0c86916bffdccd54da55d
paper_authors.name: Chenglai Zhong
paper_authors._id: 66f0c86916bffdccd54da55e
paper_authors.name: Shimin Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T07:37:06.926Z
paper_authors._id: 6522b0095e7247c2917269c0
paper_authors.avatarUrl: /avatars/746db425094a0d997446e0c0fe53a733.svg
paper_authors.fullname: Hu Shimin
paper_authors.type: user
paper_authors.user: sisyphe28
paper_authors._id: 66f0c86916bffdccd54da55f
paper_authors.name: Yudong Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:53:18.422Z
paper_authors._id: 6358a2cc1d66b442317f56c3
paper_authors.avatarUrl: /avatars/119b991a9dafefce4b9217346044f7ea.svg
paper_authors.fullname: Yudong Guo
paper_authors.type: user
paper_authors.user: YudongGuo
paper_authors._id: 66f0c86916bffdccd54da560
paper_authors.name: Juyong Zhang
paper_title: Portrait Video Editing Empowered by Multimodal Generative Priors
paper_summary: We introduce PortraitGen, a powerful portrait video editing method that
  achieves consistent and expressive stylization with multimodal prompts.
  Traditional portrait video editing methods often struggle with 3D and temporal
  consistency, and typically lack in rendering quality and efficiency. To address
  these issues, we lift the portrait video frames to a unified dynamic 3D
  Gaussian field, which ensures structural and temporal coherence across frames.
  Furthermore, we design a novel Neural Gaussian Texture mechanism that not only
  enables sophisticated style editing but also achieves rendering speed over
  100FPS. Our approach incorporates multimodal inputs through knowledge distilled
  from large-scale 2D generative models. Our system also incorporates expression
  similarity guidance and a face-aware portrait editing module, effectively
  mitigating degradation issues associated with iterative dataset updates.
  Extensive experiments demonstrate the temporal consistency, editing efficiency,
  and superior rendering quality of our method. The broad applicability of the
  proposed approach is demonstrated through various applications, including
  text-driven editing, image-driven editing, and relighting, highlighting its
  great potential to advance the field of video editing. Demo videos and released
  code are provided in our project page: https://ustc3dv.github.io/PortraitGen/
paper_ai_keywords: Gaussian field Neural Gaussian Texture structural coherence temporal coherence multimodal inputs knowledge distilled generative models expression similarity guidance face-aware portrait editing module text-driven editing image-driven editing relighting