title: Efficient Long Video Tokenization via Coordinated-based Patch
    Reconstruction
summary: Efficient tokenization of videos remains a challenge in training vision
  models that can process long videos. One promising direction is to develop a
  tokenizer that can encode long video clips, as it would enable the tokenizer to
  leverage the temporal coherence of videos better for tokenization. However,
  training existing tokenizers on long videos often incurs a huge training cost
  as they are trained to reconstruct all the frames at once. In this paper, we
  introduce CoordTok, a video tokenizer that learns a mapping from
  coordinate-based representations to the corresponding patches of input videos,
  inspired by recent advances in 3D generative models. In particular, CoordTok
  encodes a video into factorized triplane representations and reconstructs
  patches that correspond to randomly sampled (x,y,t) coordinates. This allows
  for training large tokenizer models directly on long videos without requiring
  excessive training resources. Our experiments show that CoordTok can
  drastically reduce the number of tokens for encoding long video clips. For
  instance, CoordTok can encode a 128-frame video with 128times128 resolution
  into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar
  reconstruction quality. We further show that this efficient video tokenization
  enables memory-efficient training of a diffusion transformer that can generate
  128 frames at once.
paper_authors._id: 674416d8388e030c2597afab
paper_authors.name: Huiwon Jang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:16:09.842Z
paper_authors._id: 64ae34066864362a7e290ab3
paper_authors.avatarUrl: /avatars/ae6f9fd34a697deb7ea4e92fd5f91899.svg
paper_authors.fullname: Huiwon Jang
paper_authors.type: user
paper_authors.user: huiwon
paper_authors._id: 674416d8388e030c2597afac
paper_authors.name: Sihyun Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:16:16.788Z
paper_authors._id: 64722d14609ae9f5636b6e7d
paper_authors.avatarUrl: /avatars/153447cd0b1c354a2692817a39aec5b7.svg
paper_authors.fullname: Sihyun Yu
paper_authors.type: user
paper_authors.user: sihyun-yu
paper_authors._id: 674416d8388e030c2597afad
paper_authors.name: Jinwoo Shin
paper_authors._id: 674416d8388e030c2597afae
paper_authors.name: Pieter Abbeel
paper_authors._id: 674416d8388e030c2597afaf
paper_authors.name: Younggyo Seo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T08:33:55.730Z
paper_authors._id: 6744164fe61fe2cad3ce3dc3
paper_authors.avatarUrl: /avatars/c9ffeb09f4a62bf99869b3481e0d60a3.svg
paper_authors.fullname: Younggyo Seo
paper_authors.type: user
paper_authors.user: younggyoseo
paper_title: Efficient Long Video Tokenization via Coordinated-based Patch
    Reconstruction
paper_summary: Efficient tokenization of videos remains a challenge in training vision
  models that can process long videos. One promising direction is to develop a
  tokenizer that can encode long video clips, as it would enable the tokenizer to
  leverage the temporal coherence of videos better for tokenization. However,
  training existing tokenizers on long videos often incurs a huge training cost
  as they are trained to reconstruct all the frames at once. In this paper, we
  introduce CoordTok, a video tokenizer that learns a mapping from
  coordinate-based representations to the corresponding patches of input videos,
  inspired by recent advances in 3D generative models. In particular, CoordTok
  encodes a video into factorized triplane representations and reconstructs
  patches that correspond to randomly sampled (x,y,t) coordinates. This allows
  for training large tokenizer models directly on long videos without requiring
  excessive training resources. Our experiments show that CoordTok can
  drastically reduce the number of tokens for encoding long video clips. For
  instance, CoordTok can encode a 128-frame video with 128times128 resolution
  into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar
  reconstruction quality. We further show that this efficient video tokenization
  enables memory-efficient training of a diffusion transformer that can generate
  128 frames at once.
paper_ai_summary: CoordTok, a novel video tokenizer that uses coordinate-based representations, significantly reduces the number of tokens required for long video encoding, enabling efficient training of diffusion transformers for large-scale video generation.
paper_ai_keywords: CoordTok coordinate-based representations factorized triplane representations diffusion transformer