Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts
The text-driven image and video diffusion models have achieved unprecedented
success in generating realistic and diverse content. Recently, the editing and
variation of existing images and videos in diffusion-based generative models
have garnered significant attention. However, previous works are limited to
editing content with text or providing coarse personalization using a single
visual clue, rendering them unsuitable for indescribable content that requires
fine-grained and detailed control. In this regard, we propose a generic video
editing framework called Make-A-Protagonist, which utilizes textual and visual
clues to edit videos with the goal of empowering individuals to become the
protagonists. Specifically, we leverage multiple experts to parse source video,
target visual and textual clues, and propose a visual-textual-based video
generation model that employs mask-guided denoising sampling to generate the
desired output. Extensive results demonstrate the versatile and remarkable
editing capabilities of Make-A-Protagonist.
6462d7c2d9b490218811a830
Yuyang Zhao
claimed_verified
2023-05-16T15:32:20.132Z
64638bd36c27a7e33b26654b
/avatars/2ef5aeb94ef7016082975b4cc201873e.svg
Yuyang
user
Yuyang-z
6462d7c2d9b490218811a831
Enze Xie
6462d7c2d9b490218811a832
Lanqing Hong
6462d7c2d9b490218811a833
Zhenguo Li
6462d7c2d9b490218811a834
Gim Hee Lee
claimed_verified
2023-07-24T07:44:45.439Z
64bc7568b7375f6b84503ce2
/avatars/d970c9990d9d84ac51ff225c1b26e304.svg
Gim Hee Lee
user
gimhee-lee
Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts
The text-driven image and video diffusion models have achieved unprecedented
success in generating realistic and diverse content. Recently, the editing and
variation of existing images and videos in diffusion-based generative models
have garnered significant attention. However, previous works are limited to
editing content with text or providing coarse personalization using a single
visual clue, rendering them unsuitable for indescribable content that requires
fine-grained and detailed control. In this regard, we propose a generic video
editing framework called Make-A-Protagonist, which utilizes textual and visual
clues to edit videos with the goal of empowering individuals to become the
protagonists. Specifically, we leverage multiple experts to parse source video,
target visual and textual clues, and propose a visual-textual-based video
generation model that employs mask-guided denoising sampling to generate the
desired output. Extensive results demonstrate the versatile and remarkable
editing capabilities of Make-A-Protagonist.
A video editing framework called Make-A-Protagonist uses multiple experts and mask-guided denoising sampling to edit videos with both textual and visual inputs, enabling detailed and customized content generation.
diffusion models video editing Make-A-Protagonist textual clues visual clues visual-textual-based video generation mask-guided denoising sampling