title: LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior
summary: We present LARP, a novel video tokenizer designed to overcome limitations in
  current video tokenization methods for autoregressive (AR) generative models.
  Unlike traditional patchwise tokenizers that directly encode local visual
  patches into discrete tokens, LARP introduces a holistic tokenization scheme
  that gathers information from the visual content using a set of learned
  holistic queries. This design allows LARP to capture more global and semantic
  representations, rather than being limited to local patch-level information.
  Furthermore, it offers flexibility by supporting an arbitrary number of
  discrete tokens, enabling adaptive and efficient tokenization based on the
  specific requirements of the task. To align the discrete token space with
  downstream AR generation tasks, LARP integrates a lightweight AR transformer as
  a training-time prior model that predicts the next token on its discrete latent
  space. By incorporating the prior model during training, LARP learns a latent
  space that is not only optimized for video reconstruction but is also
  structured in a way that is more conducive to autoregressive generation.
  Moreover, this process defines a sequential order for the discrete tokens,
  progressively pushing them toward an optimal configuration during training,
  ensuring smoother and more accurate AR generation at inference time.
  Comprehensive experiments demonstrate LARP's strong performance, achieving
  state-of-the-art FVD on the UCF101 class-conditional video generation
  benchmark. LARP enhances the compatibility of AR models with videos and opens
  up the potential to build unified high-fidelity multimodal large language
  models (MLLMs).
paper_authors._id: 67203e5340b7cfa09fbc4b11
paper_authors.name: Hanyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:29.660Z
paper_authors._id: 6365a174ad2c9e8b6731dd0f
paper_authors.avatarUrl: /avatars/2f4dd0eda92bca5a7464129fe7d961f9.svg
paper_authors.fullname: Hanyu Wang
paper_authors.type: user
paper_authors.user: hywang66
paper_authors._id: 67203e5340b7cfa09fbc4b12
paper_authors.name: Saksham Suri
paper_authors._id: 67203e5340b7cfa09fbc4b13
paper_authors.name: Yixuan Ren
paper_authors._id: 67203e5340b7cfa09fbc4b14
paper_authors.name: Hao Chen
paper_authors._id: 67203e5340b7cfa09fbc4b15
paper_authors.name: Abhinav Shrivastava
paper_title: LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior
paper_summary: We present LARP, a novel video tokenizer designed to overcome limitations in
  current video tokenization methods for autoregressive (AR) generative models.
  Unlike traditional patchwise tokenizers that directly encode local visual
  patches into discrete tokens, LARP introduces a holistic tokenization scheme
  that gathers information from the visual content using a set of learned
  holistic queries. This design allows LARP to capture more global and semantic
  representations, rather than being limited to local patch-level information.
  Furthermore, it offers flexibility by supporting an arbitrary number of
  discrete tokens, enabling adaptive and efficient tokenization based on the
  specific requirements of the task. To align the discrete token space with
  downstream AR generation tasks, LARP integrates a lightweight AR transformer as
  a training-time prior model that predicts the next token on its discrete latent
  space. By incorporating the prior model during training, LARP learns a latent
  space that is not only optimized for video reconstruction but is also
  structured in a way that is more conducive to autoregressive generation.
  Moreover, this process defines a sequential order for the discrete tokens,
  progressively pushing them toward an optimal configuration during training,
  ensuring smoother and more accurate AR generation at inference time.
  Comprehensive experiments demonstrate LARP's strong performance, achieving
  state-of-the-art FVD on the UCF101 class-conditional video generation
  benchmark. LARP enhances the compatibility of AR models with videos and opens
  up the potential to build unified high-fidelity multimodal large language
  models (MLLMs).
paper_ai_summary: LARP, a holistic video tokenizer with learned queries, enhances autoregressive generative models for videos through adaptive tokenization and a lightweight AR transformer, achieving state-of-the-art performance on class-conditional video generation.
paper_ai_keywords: video tokenizer autoregressive generative models holistic tokenization holistic queries global and semantic representations flexible tokenization discrete tokens lightweight AR transformer training-time prior model latent space sequential order AR generation state-of-the-art FVD UCF101 class-conditional video generation high-fidelity multimodal large language models