title: TWLV-I: Analysis and Insights from Holistic Evaluation on Video
    Foundation Models
summary: In this work, we discuss evaluating video foundation models in a fair and
  robust manner. Unlike language or image foundation models, many video
  foundation models are evaluated with differing parameters (such as sampling
  rate, number of frames, pretraining steps, etc.), making fair and robust
  comparisons challenging. Therefore, we present a carefully designed evaluation
  framework for measuring two core capabilities of video comprehension:
  appearance and motion understanding. Our findings reveal that existing video
  foundation models, whether text-supervised like UMT or InternVideo2, or
  self-supervised like V-JEPA, exhibit limitations in at least one of these
  capabilities. As an alternative, we introduce TWLV-I, a new video foundation
  model that constructs robust visual representations for both motion- and
  appearance-based videos. Based on the average top-1 accuracy of linear probing
  on five action recognition benchmarks, pretrained only on publicly accessible
  datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a
  7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger
  models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a
  2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared
  to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from
  videos of several commonly used video benchmarks, along with evaluation source
  code that can directly utilize these embeddings. The code is available on
  "https://github.com/twelvelabs-io/video-embeddings-evaluation-framework".
paper_authors._id: 66c6a6df191c0295d6cfc8c7
paper_authors.name: Hyeongmin Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:30.799Z
paper_authors._id: 64b5e71f107dfba5fd18c63a
paper_authors.avatarUrl: /avatars/5dd9c5676bbe4aa49ba1f92385a3b21f.svg
paper_authors.fullname: Hyeongmin Lee
paper_authors.type: user
paper_authors.user: Lucas-Lee
paper_authors._id: 66c6a6df191c0295d6cfc8c8
paper_authors.name: Jin-Young Kim
paper_authors._id: 66c6a6df191c0295d6cfc8c9
paper_authors.name: Kyungjune Baek
paper_authors._id: 66c6a6df191c0295d6cfc8ca
paper_authors.name: Jihwan Kim
paper_authors._id: 66c6a6df191c0295d6cfc8cb
paper_authors.name: Hyojun Go
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:09:53.339Z
paper_authors._id: 630d5aff7dacb93b3359fe26
paper_authors.avatarUrl: /avatars/e69c2bcb9d45131b68defae1bdb9d0cc.svg
paper_authors.fullname: Hyojun Go 
paper_authors.type: user
paper_authors.user: gohyojun15
paper_authors._id: 66c6a6df191c0295d6cfc8cc
paper_authors.name: Seongsu Ha
paper_authors._id: 66c6a6df191c0295d6cfc8cd
paper_authors.name: Seokjin Han
paper_authors._id: 66c6a6df191c0295d6cfc8ce
paper_authors.name: Jiho Jang
paper_authors._id: 66c6a6df191c0295d6cfc8cf
paper_authors.name: Raehyuk Jung
paper_authors._id: 66c6a6df191c0295d6cfc8d0
paper_authors.name: Daewoo Kim
paper_authors._id: 66c6a6df191c0295d6cfc8d1
paper_authors.name: GeunOh Kim
paper_authors._id: 66c6a6df191c0295d6cfc8d2
paper_authors.name: JongMok Kim
paper_authors._id: 66c6a6df191c0295d6cfc8d3
paper_authors.name: Jongseok Kim
paper_authors._id: 66c6a6df191c0295d6cfc8d4
paper_authors.name: Junwan Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:52:01.869Z
paper_authors._id: 64c2c45ae818eec6128fdda3
paper_authors.avatarUrl: /avatars/d4399e25e6399345e263c7902789047e.svg
paper_authors.fullname: Jun-Wan KIM
paper_authors.type: user
paper_authors.user: junwann
paper_authors._id: 66c6a6df191c0295d6cfc8d5
paper_authors.name: Soonwoo Kwon
paper_authors._id: 66c6a6df191c0295d6cfc8d6
paper_authors.name: Jangwon Lee
paper_authors._id: 66c6a6df191c0295d6cfc8d7
paper_authors.name: Seungjoon Park
paper_authors._id: 66c6a6df191c0295d6cfc8d8
paper_authors.name: Minjoon Seo
paper_authors._id: 66c6a6df191c0295d6cfc8d9
paper_authors.name: Jay Suh
paper_authors._id: 66c6a6df191c0295d6cfc8da
paper_authors.name: Jaehyuk Yi
paper_authors._id: 66c6a6df191c0295d6cfc8db
paper_authors.name: Aiden Lee
paper_title: TWLV-I: Analysis and Insights from Holistic Evaluation on Video
    Foundation Models
paper_summary: In this work, we discuss evaluating video foundation models in a fair and
  robust manner. Unlike language or image foundation models, many video
  foundation models are evaluated with differing parameters (such as sampling
  rate, number of frames, pretraining steps, etc.), making fair and robust
  comparisons challenging. Therefore, we present a carefully designed evaluation
  framework for measuring two core capabilities of video comprehension:
  appearance and motion understanding. Our findings reveal that existing video
  foundation models, whether text-supervised like UMT or InternVideo2, or
  self-supervised like V-JEPA, exhibit limitations in at least one of these
  capabilities. As an alternative, we introduce TWLV-I, a new video foundation
  model that constructs robust visual representations for both motion- and
  appearance-based videos. Based on the average top-1 accuracy of linear probing
  on five action recognition benchmarks, pretrained only on publicly accessible
  datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a
  7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger
  models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a
  2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared
  to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from
  videos of several commonly used video benchmarks, along with evaluation source
  code that can directly utilize these embeddings. The code is available on
  "https://github.com/twelvelabs-io/video-embeddings-evaluation-framework".
paper_ai_summary: The study introduces TWLV-I, a novel video foundation model that outperforms existing models in video comprehension by constructing robust representations for both motion and appearance.
paper_ai_keywords: video foundation models appearance understanding motion understanding linear probing action recognition benchmarks UMT InternVideo2 V-JEPA DFN embedding vectors