title: Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts
summary: The text-driven image and video diffusion models have achieved unprecedented
  success in generating realistic and diverse content. Recently, the editing and
  variation of existing images and videos in diffusion-based generative models
  have garnered significant attention. However, previous works are limited to
  editing content with text or providing coarse personalization using a single
  visual clue, rendering them unsuitable for indescribable content that requires
  fine-grained and detailed control. In this regard, we propose a generic video
  editing framework called Make-A-Protagonist, which utilizes textual and visual
  clues to edit videos with the goal of empowering individuals to become the
  protagonists. Specifically, we leverage multiple experts to parse source video,
  target visual and textual clues, and propose a visual-textual-based video
  generation model that employs mask-guided denoising sampling to generate the
  desired output. Extensive results demonstrate the versatile and remarkable
  editing capabilities of Make-A-Protagonist.
paper_authors._id: 6462d7c2d9b490218811a830
paper_authors.name: Yuyang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:32:20.132Z
paper_authors._id: 64638bd36c27a7e33b26654b
paper_authors.avatarUrl: /avatars/2ef5aeb94ef7016082975b4cc201873e.svg
paper_authors.fullname: Yuyang
paper_authors.type: user
paper_authors.user: Yuyang-z
paper_authors._id: 6462d7c2d9b490218811a831
paper_authors.name: Enze Xie
paper_authors._id: 6462d7c2d9b490218811a832
paper_authors.name: Lanqing Hong
paper_authors._id: 6462d7c2d9b490218811a833
paper_authors.name: Zhenguo Li
paper_authors._id: 6462d7c2d9b490218811a834
paper_authors.name: Gim Hee Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-24T07:44:45.439Z
paper_authors._id: 64bc7568b7375f6b84503ce2
paper_authors.avatarUrl: /avatars/d970c9990d9d84ac51ff225c1b26e304.svg
paper_authors.fullname: Gim Hee Lee
paper_authors.type: user
paper_authors.user: gimhee-lee
paper_title: Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts
paper_summary: The text-driven image and video diffusion models have achieved unprecedented
  success in generating realistic and diverse content. Recently, the editing and
  variation of existing images and videos in diffusion-based generative models
  have garnered significant attention. However, previous works are limited to
  editing content with text or providing coarse personalization using a single
  visual clue, rendering them unsuitable for indescribable content that requires
  fine-grained and detailed control. In this regard, we propose a generic video
  editing framework called Make-A-Protagonist, which utilizes textual and visual
  clues to edit videos with the goal of empowering individuals to become the
  protagonists. Specifically, we leverage multiple experts to parse source video,
  target visual and textual clues, and propose a visual-textual-based video
  generation model that employs mask-guided denoising sampling to generate the
  desired output. Extensive results demonstrate the versatile and remarkable
  editing capabilities of Make-A-Protagonist.
paper_ai_keywords: diffusion models text-driven image text-driven video synthesis editing variation diffusion-based generative models textual clues visual clues fine-grained detailed control video editing framework mask-guided denoising sampling visual-textual-based video generation model