title: TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
summary: Within recent approaches to text-to-video (T2V) generation, achieving
  controllability in the synthesized video is often a challenge. Typically, this
  issue is addressed by providing low-level per-frame guidance in the form of
  edge maps, depth maps, or an existing video to be altered. However, the process
  of obtaining such guidance can be labor-intensive. This paper focuses on
  enhancing controllability in video synthesis by employing straightforward
  bounding boxes to guide the subject in various ways, all without the need for
  neural network training, finetuning, optimization at inference time, or the use
  of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a
  pre-trained (T2V) model, and easy to implement. The subject is directed by a
  bounding box through the proposed spatial and temporal attention map editing.
  Moreover, we introduce the concept of keyframing, allowing the subject
  trajectory and overall appearance to be guided by both a moving bounding box
  and corresponding prompts, without the need to provide a detailed mask. The
  method is efficient, with negligible additional computation relative to the
  underlying pre-trained model. Despite the simplicity of the bounding box
  guidance, the resulting motion is surprisingly natural, with emergent effects
  including perspective and movement toward the virtual camera as the box size
  increases.
paper_authors._id: 6594bba6c0a2664426a96d7c
paper_authors.name: Wan-Duo Kurt Ma
paper_authors._id: 6594bba6c0a2664426a96d7d
paper_authors.name: J. P. Lewis
paper_authors._id: 6594bba6c0a2664426a96d7e
paper_authors.name: W. Bastiaan Kleijn
paper_title: TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
paper_summary: Within recent approaches to text-to-video (T2V) generation, achieving
  controllability in the synthesized video is often a challenge. Typically, this
  issue is addressed by providing low-level per-frame guidance in the form of
  edge maps, depth maps, or an existing video to be altered. However, the process
  of obtaining such guidance can be labor-intensive. This paper focuses on
  enhancing controllability in video synthesis by employing straightforward
  bounding boxes to guide the subject in various ways, all without the need for
  neural network training, finetuning, optimization at inference time, or the use
  of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a
  pre-trained (T2V) model, and easy to implement. The subject is directed by a
  bounding box through the proposed spatial and temporal attention map editing.
  Moreover, we introduce the concept of keyframing, allowing the subject
  trajectory and overall appearance to be guided by both a moving bounding box
  and corresponding prompts, without the need to provide a detailed mask. The
  method is efficient, with negligible additional computation relative to the
  underlying pre-trained model. Despite the simplicity of the bounding box
  guidance, the resulting motion is surprisingly natural, with emergent effects
  including perspective and movement toward the virtual camera as the box size
  increases.
paper_ai_keywords: T2V generation bounding boxes pre-trained model spatial attention map temporal attention map keyframing