title: VidTok: A Versatile and Open-Source Video Tokenizer
summary: Encoding video content into compact latent tokens has become a fundamental
  step in video generation and understanding, driven by the need to address the
  inherent redundancy in pixel-level representations. Consequently, there is a
  growing demand for high-performance, open-source video tokenizers as
  video-centric research gains prominence. We introduce VidTok, a versatile video
  tokenizer that delivers state-of-the-art performance in both continuous and
  discrete tokenizations. VidTok incorporates several key advancements over
  existing approaches: 1) model architecture such as convolutional layers and
  up/downsampling modules; 2) to address the training instability and codebook
  collapse commonly associated with conventional Vector Quantization (VQ), we
  integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3)
  improved training strategies, including a two-stage training process and the
  use of reduced frame rates. By integrating these advancements, VidTok achieves
  substantial improvements over existing methods, demonstrating superior
  performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD,
  under standardized evaluation settings.
paper_authors._id: 6762c51eeceb9f88f41d1c23
paper_authors.name: Anni Tang
paper_authors._id: 6762c51eeceb9f88f41d1c24
paper_authors.name: Tianyu He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:58.814Z
paper_authors._id: 619b7b1cab4c7b7f16a7d59e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
paper_authors.fullname: Tianyu He
paper_authors.type: user
paper_authors.user: deeptimhe
paper_authors._id: 6762c51eeceb9f88f41d1c25
paper_authors.name: Junliang Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:14:02.510Z
paper_authors._id: 66868659ccb9539da85c4e14
paper_authors.avatarUrl: /avatars/515a49363872c23d57a6f75063606348.svg
paper_authors.fullname: Junliang Guo
paper_authors.type: user
paper_authors.user: leo-guo
paper_authors._id: 6762c51eeceb9f88f41d1c26
paper_authors.name: Xinle Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:13:56.564Z
paper_authors._id: 657ab561270ef0b785f21135
paper_authors.avatarUrl: /avatars/6a153e8e4d93e32d11d4f537e0159ccf.svg
paper_authors.fullname: Xinle Cheng
paper_authors.type: user
paper_authors.user: CXL295
paper_authors._id: 6762c51eeceb9f88f41d1c27
paper_authors.name: Li Song
paper_authors._id: 6762c51eeceb9f88f41d1c28
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:12:54.696Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_title: VidTok: A Versatile and Open-Source Video Tokenizer
paper_summary: Encoding video content into compact latent tokens has become a fundamental
  step in video generation and understanding, driven by the need to address the
  inherent redundancy in pixel-level representations. Consequently, there is a
  growing demand for high-performance, open-source video tokenizers as
  video-centric research gains prominence. We introduce VidTok, a versatile video
  tokenizer that delivers state-of-the-art performance in both continuous and
  discrete tokenizations. VidTok incorporates several key advancements over
  existing approaches: 1) model architecture such as convolutional layers and
  up/downsampling modules; 2) to address the training instability and codebook
  collapse commonly associated with conventional Vector Quantization (VQ), we
  integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3)
  improved training strategies, including a two-stage training process and the
  use of reduced frame rates. By integrating these advancements, VidTok achieves
  substantial improvements over existing methods, demonstrating superior
  performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD,
  under standardized evaluation settings.
paper_ai_keywords: VidTok convolutional layers up/downsampling modules Vector Quantization (VQ) Finite Scalar Quantization (FSQ) PSNR SSIM LPIPS FVD