title: Sketching the Future (STF): Applying Conditional Control Techniques to
    Text-to-Video Models
summary: The proliferation of video content demands efficient and flexible neural
  network based approaches for generating new video content. In this paper, we
  propose a novel approach that combines zero-shot text-to-video generation with
  ControlNet to improve the output of these models. Our method takes multiple
  sketched frames as input and generates video output that matches the flow of
  these frames, building upon the Text-to-Video Zero architecture and
  incorporating ControlNet to enable additional input conditions. By first
  interpolating frames between the inputted sketches and then running
  Text-to-Video Zero using the new interpolated frames video as the control
  technique, we leverage the benefits of both zero-shot text-to-video generation
  and the robust control provided by ControlNet. Experiments demonstrate that our
  method excels at producing high-quality and remarkably consistent video content
  that more accurately aligns with the user's intended motion for the subject
  within the video. We provide a comprehensive resource package, including a demo
  video, project website, open-source GitHub repository, and a Colab playground
  to foster further research and application of our proposed method.
paper_authors._id: 645c4213fbdc26a155fc9b1d
paper_authors.name: Rohan Dhesikan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:15:32.439Z
paper_authors._id: 6446ce6615a27291ef8c437e
paper_authors.avatarUrl: /avatars/e9f4f5d2cfe52fd8458930b3ab9e69e4.svg
paper_authors.fullname: Rohan Dhesikan
paper_authors.type: user
paper_authors.user: rohandkn
paper_authors._id: 645c4213fbdc26a155fc9b1e
paper_authors.name: Vignesh Rajmohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:13:47.971Z
paper_authors._id: 6445fb862d91b15b4c825255
paper_authors.avatarUrl: /avatars/b6b44af2be6a6e2057a25872aeca9427.svg
paper_authors.fullname: Vignesh Rajmohan
paper_authors.type: user
paper_authors.user: vigneshrajmohan
paper_title: Sketching the Future (STF): Applying Conditional Control Techniques to
    Text-to-Video Models
paper_summary: The proliferation of video content demands efficient and flexible neural
  network based approaches for generating new video content. In this paper, we
  propose a novel approach that combines zero-shot text-to-video generation with
  ControlNet to improve the output of these models. Our method takes multiple
  sketched frames as input and generates video output that matches the flow of
  these frames, building upon the Text-to-Video Zero architecture and
  incorporating ControlNet to enable additional input conditions. By first
  interpolating frames between the inputted sketches and then running
  Text-to-Video Zero using the new interpolated frames video as the control
  technique, we leverage the benefits of both zero-shot text-to-video generation
  and the robust control provided by ControlNet. Experiments demonstrate that our
  method excels at producing high-quality and remarkably consistent video content
  that more accurately aligns with the user's intended motion for the subject
  within the video. We provide a comprehensive resource package, including a demo
  video, project website, open-source GitHub repository, and a Colab playground
  to foster further research and application of our proposed method.
paper_ai_summary: A novel method combines zero-shot text-to-video generation with ControlNet for improved video content creation, demonstrating high-quality and consistent video outputs.
paper_ai_keywords: zero-shot text-to-video generation ControlNet interpolated frames Text-to-Video Zero control technique video content high-quality consistent user's intended motion