InteractiveVideo: User-Centric Controllable Video Generation with
  Synergistic Multimodal Instructions
We introduce InteractiveVideo, a user-centric framework for video
generation. Different from traditional generative approaches that operate based
on user-provided images or text, our framework is designed for dynamic
interaction, allowing users to instruct the generative model through various
intuitive mechanisms during the whole generation process, e.g. text and image
prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal
Instruction mechanism, designed to seamlessly integrate users' multimodal
instructions into generative models, thus facilitating a cooperative and
responsive interaction between user inputs and the generative process. This
approach enables iterative and fine-grained refinement of the generation result
through precise and effective user instructions. With
InteractiveVideo, users are given the flexibility to meticulously
tailor key aspects of a video. They can paint the reference image, edit
semantics, and adjust video motions until their requirements are fully met.
Code, models, and demo are available at
https://github.com/invictus717/InteractiveVideo
65c19e934f21259d99369c6f
Yiyuan Zhang
claimed_verified
2024-02-07T20:41:10.184Z
63176933b58b0184630d2c74
https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
Yiyuan Zhang
user
Yiyuan
65c19e934f21259d99369c70
Yuhao Kang
65c19e934f21259d99369c71
Zhixin Zhang
65c19e934f21259d99369c72
Xiaohan Ding
65c19e934f21259d99369c73
Sanyuan Zhao
65c19e934f21259d99369c74
Xiangyu Yue
InteractiveVideo: User-Centric Controllable Video Generation with
  Synergistic Multimodal Instructions
We introduce InteractiveVideo, a user-centric framework for video
generation. Different from traditional generative approaches that operate based
on user-provided images or text, our framework is designed for dynamic
interaction, allowing users to instruct the generative model through various
intuitive mechanisms during the whole generation process, e.g. text and image
prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal
Instruction mechanism, designed to seamlessly integrate users' multimodal
instructions into generative models, thus facilitating a cooperative and
responsive interaction between user inputs and the generative process. This
approach enables iterative and fine-grained refinement of the generation result
through precise and effective user instructions. With
InteractiveVideo, users are given the flexibility to meticulously
tailor key aspects of a video. They can paint the reference image, edit
semantics, and adjust video motions until their requirements are fully met.
Code, models, and demo are available at
https://github.com/invictus717/InteractiveVideo
InteractiveVideo is a user-centric framework enabling dynamic interaction and fine-grained refinement of video generation through multimodal inputs.
Synergistic Multimodal Instruction mechanism