title: Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
summary: In recent years, there has been a significant surge of interest in unifying
  image comprehension and generation within Large Language Models (LLMs). This
  growing interest has prompted us to explore extending this unification to
  videos. The core challenge lies in developing a versatile video tokenizer that
  captures both the spatial characteristics and temporal dynamics of videos to
  obtain representations for LLMs, and the representations can be further decoded
  into realistic video clips to enable video generation. In this work, we
  introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the
  diffusion process for self-supervised video representation learning. We posit
  that if a video diffusion model can effectively de-noise video clips by taking
  the features of a video tokenizer as the condition, then the tokenizer has
  successfully captured robust spatial and temporal information. Additionally,
  the video diffusion model inherently functions as a de-tokenizer, decoding
  videos from their representations. Building upon the Divot tokenizer, we
  present Divot-Vicuna through video-to-text autoregression and text-to-video
  generation by modeling the distributions of continuous-valued Divot features
  with a Gaussian Mixture Model. Experimental results demonstrate that our
  diffusion-based video tokenizer, when integrated with a pre-trained LLM,
  achieves competitive performance across various video comprehension and
  generation benchmarks. The instruction tuned Divot-Vicuna also excels in video
  storytelling, generating interleaved narratives and corresponding videos.
paper_authors._id: 675260813c0a3886978b2e48
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:42.443Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 675260813c0a3886978b2e49
paper_authors.name: Yizhuo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:48.137Z
paper_authors._id: 630b094f8b327c7b8b94d24c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630b094f8b327c7b8b94d24c/Fd0ugLOHJZIi8oUZScOmX.jpeg
paper_authors.fullname: Yizhuo Li
paper_authors.type: user
paper_authors.user: liyz
paper_authors._id: 675260813c0a3886978b2e4a
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:54.425Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 675260813c0a3886978b2e4b
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:17:01.613Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
paper_summary: In recent years, there has been a significant surge of interest in unifying
  image comprehension and generation within Large Language Models (LLMs). This
  growing interest has prompted us to explore extending this unification to
  videos. The core challenge lies in developing a versatile video tokenizer that
  captures both the spatial characteristics and temporal dynamics of videos to
  obtain representations for LLMs, and the representations can be further decoded
  into realistic video clips to enable video generation. In this work, we
  introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the
  diffusion process for self-supervised video representation learning. We posit
  that if a video diffusion model can effectively de-noise video clips by taking
  the features of a video tokenizer as the condition, then the tokenizer has
  successfully captured robust spatial and temporal information. Additionally,
  the video diffusion model inherently functions as a de-tokenizer, decoding
  videos from their representations. Building upon the Divot tokenizer, we
  present Divot-Vicuna through video-to-text autoregression and text-to-video
  generation by modeling the distributions of continuous-valued Divot features
  with a Gaussian Mixture Model. Experimental results demonstrate that our
  diffusion-based video tokenizer, when integrated with a pre-trained LLM,
  achieves competitive performance across various video comprehension and
  generation benchmarks. The instruction tuned Divot-Vicuna also excels in video
  storytelling, generating interleaved narratives and corresponding videos.
paper_ai_summary: A diffusion-powered video tokenizer (Divot) captures spatial and temporal video features for integration with LLMs, enabling video comprehension and generation with competitive performance.
paper_ai_keywords: diffusion-Powered Video Tokenizer diffusion process self-supervised video representation learning video diffusion model Gaussian Mixture Model video-to-text autoregression text-to-video generation video storytelling