title: Packing Input Frame Context in Next-Frame Prediction Models for Video
    Generation
summary: We present a neural network structure, FramePack, to train next-frame (or
  next-frame-section) prediction models for video generation. The FramePack
  compresses input frames to make the transformer context length a fixed number
  regardless of the video length. As a result, we are able to process a large
  number of frames using video diffusion with computation bottleneck similar to
  image diffusion. This also makes the training video batch sizes significantly
  higher (batch sizes become comparable to image diffusion training). We also
  propose an anti-drifting sampling method that generates frames in inverted
  temporal order with early-established endpoints to avoid exposure bias (error
  accumulation over iterations). Finally, we show that existing video diffusion
  models can be finetuned with FramePack, and their visual quality may be
  improved because the next-frame prediction supports more balanced diffusion
  schedulers with less extreme flow shift timesteps.
paper_authors._id: 6801b65181552de84a4b7e29
paper_authors.name: Lvmin Zhang
paper_authors._id: 6801b65181552de84a4b7e2a
paper_authors.name: Maneesh Agrawala
paper_title: Packing Input Frame Context in Next-Frame Prediction Models for Video
    Generation
paper_summary: We present a neural network structure, FramePack, to train next-frame (or
  next-frame-section) prediction models for video generation. The FramePack
  compresses input frames to make the transformer context length a fixed number
  regardless of the video length. As a result, we are able to process a large
  number of frames using video diffusion with computation bottleneck similar to
  image diffusion. This also makes the training video batch sizes significantly
  higher (batch sizes become comparable to image diffusion training). We also
  propose an anti-drifting sampling method that generates frames in inverted
  temporal order with early-established endpoints to avoid exposure bias (error
  accumulation over iterations). Finally, we show that existing video diffusion
  models can be finetuned with FramePack, and their visual quality may be
  improved because the next-frame prediction supports more balanced diffusion
  schedulers with less extreme flow shift timesteps.
paper_ai_summary: FramePack, a neural network for video generation, compresses frames to manage transformer context length and enhances video diffusion models with increased batch sizes and improved frame prediction.
paper_ai_keywords: neural network FramePack next-frame prediction transformer context length video diffusion computation bottleneck image diffusion batch size anti-drifting sampling inverted temporal order exposure bias diffusion schedulers flow shift