Mind the Time: Temporally-Controlled Multi-Event Video Generation
Real-world videos consist of sequences of events. Generating such sequences
with precise temporal control is infeasible with existing video generators that
rely on a single paragraph of text as input. When tasked with generating
multiple events described using a single prompt, such methods often ignore some
of the events or fail to arrange them in the correct order. To address this
limitation, we present MinT, a multi-event video generator with temporal
control. Our key insight is to bind each event to a specific period in the
generated video, which allows the model to focus on one event at a time. To
enable time-aware interactions between event captions and video tokens, we
design a time-based positional encoding method, dubbed ReRoPE. This encoding
helps to guide the cross-attention operation. By fine-tuning a pre-trained
video diffusion transformer on temporally grounded data, our approach produces
coherent videos with smoothly connected events. For the first time in the
literature, our model offers control over the timing of events in generated
videos. Extensive experiments demonstrate that MinT outperforms existing
open-source models by a large margin.
67567ea95b91dea24c1901cd
Ziyi Wu
admin_assigned
2024-12-09T16:00:38.322Z
62980664ff0acd7e027d6686
/avatars/364d4c8432c24775a099641fc576dbdc.svg
Ziyi Wu
user
Dazitu616
67567ea95b91dea24c1901ce
Aliaksandr Siarohin
admin_assigned
2024-12-09T15:59:44.591Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
67567ea95b91dea24c1901cf
Willi Menapace
admin_assigned
2024-12-09T15:59:36.298Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
67567ea95b91dea24c1901d0
Ivan Skorokhodov
admin_assigned
2024-12-09T15:59:27.509Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
67567ea95b91dea24c1901d1
Yuwei Fang
admin_assigned
2024-12-09T15:59:21.018Z
6406d1dcc3ab325efa9db198
/avatars/781f3cd455596f7cffa0906cc487f79b.svg
Yuwei Fang
user
studyfang
67567ea95b91dea24c1901d2
Varnith Chordia
admin_assigned
2024-12-09T15:59:06.392Z
650b1d482ec41a31f30429c7
/avatars/1227d6f168646248b1c10dd66624e23a.svg
Varnith Chordia
user
vchordia
67567ea95b91dea24c1901d3
Igor Gilitschenski
admin_assigned
2024-12-09T15:59:01.077Z
64893f0b9ab28735f1a1ec0b
https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
Igor Gilitschenski
user
igilitschenski
67567ea95b91dea24c1901d4
Sergey Tulyakov
Mind the Time: Temporally-Controlled Multi-Event Video Generation
Real-world videos consist of sequences of events. Generating such sequences
with precise temporal control is infeasible with existing video generators that
rely on a single paragraph of text as input. When tasked with generating
multiple events described using a single prompt, such methods often ignore some
of the events or fail to arrange them in the correct order. To address this
limitation, we present MinT, a multi-event video generator with temporal
control. Our key insight is to bind each event to a specific period in the
generated video, which allows the model to focus on one event at a time. To
enable time-aware interactions between event captions and video tokens, we
design a time-based positional encoding method, dubbed ReRoPE. This encoding
helps to guide the cross-attention operation. By fine-tuning a pre-trained
video diffusion transformer on temporally grounded data, our approach produces
coherent videos with smoothly connected events. For the first time in the
literature, our model offers control over the timing of events in generated
videos. Extensive experiments demonstrate that MinT outperforms existing
open-source models by a large margin.
MinT is a multi-event video generator that controls temporal sequences by binding events to specific periods and using a time-based positional encoding to guide cross-attention in a fine-tuned video diffusion transformer.
multi-event video generator temporal control event captions video tokens ReRoPE time-based positional encoding cross-attention video diffusion transformer temporally grounded data coherent videos