title: InteractiveVideo: User-Centric Controllable Video Generation with
    Synergistic Multimodal Instructions
summary: We introduce InteractiveVideo, a user-centric framework for video
  generation. Different from traditional generative approaches that operate based
  on user-provided images or text, our framework is designed for dynamic
  interaction, allowing users to instruct the generative model through various
  intuitive mechanisms during the whole generation process, e.g. text and image
  prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal
  Instruction mechanism, designed to seamlessly integrate users' multimodal
  instructions into generative models, thus facilitating a cooperative and
  responsive interaction between user inputs and the generative process. This
  approach enables iterative and fine-grained refinement of the generation result
  through precise and effective user instructions. With
  InteractiveVideo, users are given the flexibility to meticulously
  tailor key aspects of a video. They can paint the reference image, edit
  semantics, and adjust video motions until their requirements are fully met.
  Code, models, and demo are available at
  https://github.com/invictus717/InteractiveVideo
paper_authors._id: 65c19e934f21259d99369c6f
paper_authors.name: Yiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T20:41:10.184Z
paper_authors._id: 63176933b58b0184630d2c74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
paper_authors.fullname: Yiyuan Zhang
paper_authors.type: user
paper_authors.user: Yiyuan
paper_authors._id: 65c19e934f21259d99369c70
paper_authors.name: Yuhao Kang
paper_authors._id: 65c19e934f21259d99369c71
paper_authors.name: Zhixin Zhang
paper_authors._id: 65c19e934f21259d99369c72
paper_authors.name: Xiaohan Ding
paper_authors._id: 65c19e934f21259d99369c73
paper_authors.name: Sanyuan Zhao
paper_authors._id: 65c19e934f21259d99369c74
paper_authors.name: Xiangyu Yue
paper_title: InteractiveVideo: User-Centric Controllable Video Generation with
    Synergistic Multimodal Instructions
paper_summary: We introduce InteractiveVideo, a user-centric framework for video
  generation. Different from traditional generative approaches that operate based
  on user-provided images or text, our framework is designed for dynamic
  interaction, allowing users to instruct the generative model through various
  intuitive mechanisms during the whole generation process, e.g. text and image
  prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal
  Instruction mechanism, designed to seamlessly integrate users' multimodal
  instructions into generative models, thus facilitating a cooperative and
  responsive interaction between user inputs and the generative process. This
  approach enables iterative and fine-grained refinement of the generation result
  through precise and effective user instructions. With
  InteractiveVideo, users are given the flexibility to meticulously
  tailor key aspects of a video. They can paint the reference image, edit
  semantics, and adjust video motions until their requirements are fully met.
  Code, models, and demo are available at
  https://github.com/invictus717/InteractiveVideo
paper_ai_summary: InteractiveVideo is a user-centric framework enabling dynamic interaction and fine-grained refinement of video generation through multimodal inputs.
paper_ai_keywords: Synergistic Multimodal Instruction mechanism