Efficient Long Video Tokenization via Coordinated-based Patch
  Reconstruction
Efficient tokenization of videos remains a challenge in training vision
models that can process long videos. One promising direction is to develop a
tokenizer that can encode long video clips, as it would enable the tokenizer to
leverage the temporal coherence of videos better for tokenization. However,
training existing tokenizers on long videos often incurs a huge training cost
as they are trained to reconstruct all the frames at once. In this paper, we
introduce CoordTok, a video tokenizer that learns a mapping from
coordinate-based representations to the corresponding patches of input videos,
inspired by recent advances in 3D generative models. In particular, CoordTok
encodes a video into factorized triplane representations and reconstructs
patches that correspond to randomly sampled (x,y,t) coordinates. This allows
for training large tokenizer models directly on long videos without requiring
excessive training resources. Our experiments show that CoordTok can
drastically reduce the number of tokens for encoding long video clips. For
instance, CoordTok can encode a 128-frame video with 128times128 resolution
into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar
reconstruction quality. We further show that this efficient video tokenization
enables memory-efficient training of a diffusion transformer that can generate
128 frames at once.
674416d8388e030c2597afab
Huiwon Jang
admin_assigned
2024-11-25T21:16:09.842Z
64ae34066864362a7e290ab3
/avatars/ae6f9fd34a697deb7ea4e92fd5f91899.svg
Huiwon Jang
user
huiwon
674416d8388e030c2597afac
Sihyun Yu
admin_assigned
2024-11-25T21:16:16.788Z
64722d14609ae9f5636b6e7d
/avatars/153447cd0b1c354a2692817a39aec5b7.svg
Sihyun Yu
user
sihyun-yu
674416d8388e030c2597afad
Jinwoo Shin
674416d8388e030c2597afae
Pieter Abbeel
674416d8388e030c2597afaf
Younggyo Seo
claimed_verified
2024-11-25T08:33:55.730Z
6744164fe61fe2cad3ce3dc3
/avatars/c9ffeb09f4a62bf99869b3481e0d60a3.svg
Younggyo Seo
user
younggyoseo
Efficient Long Video Tokenization via Coordinated-based Patch
  Reconstruction
Efficient tokenization of videos remains a challenge in training vision
models that can process long videos. One promising direction is to develop a
tokenizer that can encode long video clips, as it would enable the tokenizer to
leverage the temporal coherence of videos better for tokenization. However,
training existing tokenizers on long videos often incurs a huge training cost
as they are trained to reconstruct all the frames at once. In this paper, we
introduce CoordTok, a video tokenizer that learns a mapping from
coordinate-based representations to the corresponding patches of input videos,
inspired by recent advances in 3D generative models. In particular, CoordTok
encodes a video into factorized triplane representations and reconstructs
patches that correspond to randomly sampled (x,y,t) coordinates. This allows
for training large tokenizer models directly on long videos without requiring
excessive training resources. Our experiments show that CoordTok can
drastically reduce the number of tokens for encoding long video clips. For
instance, CoordTok can encode a 128-frame video with 128times128 resolution
into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar
reconstruction quality. We further show that this efficient video tokenization
enables memory-efficient training of a diffusion transformer that can generate
128 frames at once.
CoordTok, a novel video tokenizer that uses coordinate-based representations, significantly reduces the number of tokens required for long video encoding, enabling efficient training of diffusion transformers for large-scale video generation.
CoordTok coordinate-based representations factorized triplane representations diffusion transformer