Sketching the Future (STF): Applying Conditional Control Techniques to
  Text-to-Video Models
The proliferation of video content demands efficient and flexible neural
network based approaches for generating new video content. In this paper, we
propose a novel approach that combines zero-shot text-to-video generation with
ControlNet to improve the output of these models. Our method takes multiple
sketched frames as input and generates video output that matches the flow of
these frames, building upon the Text-to-Video Zero architecture and
incorporating ControlNet to enable additional input conditions. By first
interpolating frames between the inputted sketches and then running
Text-to-Video Zero using the new interpolated frames video as the control
technique, we leverage the benefits of both zero-shot text-to-video generation
and the robust control provided by ControlNet. Experiments demonstrate that our
method excels at producing high-quality and remarkably consistent video content
that more accurately aligns with the user's intended motion for the subject
within the video. We provide a comprehensive resource package, including a demo
video, project website, open-source GitHub repository, and a Colab playground
to foster further research and application of our proposed method.
645c4213fbdc26a155fc9b1d
Rohan Dhesikan
admin_assigned
2023-05-24T09:15:32.439Z
6446ce6615a27291ef8c437e
/avatars/e9f4f5d2cfe52fd8458930b3ab9e69e4.svg
Rohan Dhesikan
user
rohandkn
645c4213fbdc26a155fc9b1e
Vignesh Rajmohan
admin_assigned
2023-05-24T09:13:47.971Z
6445fb862d91b15b4c825255
/avatars/b6b44af2be6a6e2057a25872aeca9427.svg
Vignesh Rajmohan
user
vigneshrajmohan
Sketching the Future (STF): Applying Conditional Control Techniques to
  Text-to-Video Models
The proliferation of video content demands efficient and flexible neural
network based approaches for generating new video content. In this paper, we
propose a novel approach that combines zero-shot text-to-video generation with
ControlNet to improve the output of these models. Our method takes multiple
sketched frames as input and generates video output that matches the flow of
these frames, building upon the Text-to-Video Zero architecture and
incorporating ControlNet to enable additional input conditions. By first
interpolating frames between the inputted sketches and then running
Text-to-Video Zero using the new interpolated frames video as the control
technique, we leverage the benefits of both zero-shot text-to-video generation
and the robust control provided by ControlNet. Experiments demonstrate that our
method excels at producing high-quality and remarkably consistent video content
that more accurately aligns with the user's intended motion for the subject
within the video. We provide a comprehensive resource package, including a demo
video, project website, open-source GitHub repository, and a Colab playground
to foster further research and application of our proposed method.
A novel method combines zero-shot text-to-video generation with ControlNet for improved video content creation, demonstrating high-quality and consistent video outputs.
zero-shot text-to-video generation ControlNet interpolated frames Text-to-Video Zero control technique video content high-quality consistent user's intended motion