title: Mind the Time: Temporally-Controlled Multi-Event Video Generation
summary: Real-world videos consist of sequences of events. Generating such sequences
  with precise temporal control is infeasible with existing video generators that
  rely on a single paragraph of text as input. When tasked with generating
  multiple events described using a single prompt, such methods often ignore some
  of the events or fail to arrange them in the correct order. To address this
  limitation, we present MinT, a multi-event video generator with temporal
  control. Our key insight is to bind each event to a specific period in the
  generated video, which allows the model to focus on one event at a time. To
  enable time-aware interactions between event captions and video tokens, we
  design a time-based positional encoding method, dubbed ReRoPE. This encoding
  helps to guide the cross-attention operation. By fine-tuning a pre-trained
  video diffusion transformer on temporally grounded data, our approach produces
  coherent videos with smoothly connected events. For the first time in the
  literature, our model offers control over the timing of events in generated
  videos. Extensive experiments demonstrate that MinT outperforms existing
  open-source models by a large margin.
paper_authors._id: 67567ea95b91dea24c1901cd
paper_authors.name: Ziyi Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T16:00:38.322Z
paper_authors._id: 62980664ff0acd7e027d6686
paper_authors.avatarUrl: /avatars/364d4c8432c24775a099641fc576dbdc.svg
paper_authors.fullname: Ziyi Wu
paper_authors.type: user
paper_authors.user: Dazitu616
paper_authors._id: 67567ea95b91dea24c1901ce
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:59:44.591Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 67567ea95b91dea24c1901cf
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:59:36.298Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 67567ea95b91dea24c1901d0
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:59:27.509Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 67567ea95b91dea24c1901d1
paper_authors.name: Yuwei Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:59:21.018Z
paper_authors._id: 6406d1dcc3ab325efa9db198
paper_authors.avatarUrl: /avatars/781f3cd455596f7cffa0906cc487f79b.svg
paper_authors.fullname: Yuwei Fang
paper_authors.type: user
paper_authors.user: studyfang
paper_authors._id: 67567ea95b91dea24c1901d2
paper_authors.name: Varnith Chordia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:59:06.392Z
paper_authors._id: 650b1d482ec41a31f30429c7
paper_authors.avatarUrl: /avatars/1227d6f168646248b1c10dd66624e23a.svg
paper_authors.fullname: Varnith Chordia
paper_authors.type: user
paper_authors.user: vchordia
paper_authors._id: 67567ea95b91dea24c1901d3
paper_authors.name: Igor Gilitschenski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:59:01.077Z
paper_authors._id: 64893f0b9ab28735f1a1ec0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
paper_authors.fullname: Igor Gilitschenski
paper_authors.type: user
paper_authors.user: igilitschenski
paper_authors._id: 67567ea95b91dea24c1901d4
paper_authors.name: Sergey Tulyakov
paper_title: Mind the Time: Temporally-Controlled Multi-Event Video Generation
paper_summary: Real-world videos consist of sequences of events. Generating such sequences
  with precise temporal control is infeasible with existing video generators that
  rely on a single paragraph of text as input. When tasked with generating
  multiple events described using a single prompt, such methods often ignore some
  of the events or fail to arrange them in the correct order. To address this
  limitation, we present MinT, a multi-event video generator with temporal
  control. Our key insight is to bind each event to a specific period in the
  generated video, which allows the model to focus on one event at a time. To
  enable time-aware interactions between event captions and video tokens, we
  design a time-based positional encoding method, dubbed ReRoPE. This encoding
  helps to guide the cross-attention operation. By fine-tuning a pre-trained
  video diffusion transformer on temporally grounded data, our approach produces
  coherent videos with smoothly connected events. For the first time in the
  literature, our model offers control over the timing of events in generated
  videos. Extensive experiments demonstrate that MinT outperforms existing
  open-source models by a large margin.
paper_ai_summary: MinT is a multi-event video generator that controls temporal sequences by binding events to specific periods and using a time-based positional encoding to guide cross-attention in a fine-tuned video diffusion transformer.
paper_ai_keywords: multi-event video generator temporal control event captions video tokens ReRoPE time-based positional encoding cross-attention video diffusion transformer temporally grounded data coherent videos