TWLV-I: Analysis and Insights from Holistic Evaluation on Video
  Foundation Models
In this work, we discuss evaluating video foundation models in a fair and
robust manner. Unlike language or image foundation models, many video
foundation models are evaluated with differing parameters (such as sampling
rate, number of frames, pretraining steps, etc.), making fair and robust
comparisons challenging. Therefore, we present a carefully designed evaluation
framework for measuring two core capabilities of video comprehension:
appearance and motion understanding. Our findings reveal that existing video
foundation models, whether text-supervised like UMT or InternVideo2, or
self-supervised like V-JEPA, exhibit limitations in at least one of these
capabilities. As an alternative, we introduce TWLV-I, a new video foundation
model that constructs robust visual representations for both motion- and
appearance-based videos. Based on the average top-1 accuracy of linear probing
on five action recognition benchmarks, pretrained only on publicly accessible
datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a
7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger
models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a
2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared
to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from
videos of several commonly used video benchmarks, along with evaluation source
code that can directly utilize these embeddings. The code is available on
"https://github.com/twelvelabs-io/video-embeddings-evaluation-framework".
66c6a6df191c0295d6cfc8c7
Hyeongmin Lee
claimed_verified
2024-08-22T19:50:30.799Z
64b5e71f107dfba5fd18c63a
/avatars/5dd9c5676bbe4aa49ba1f92385a3b21f.svg
Hyeongmin Lee
user
Lucas-Lee
66c6a6df191c0295d6cfc8c8
Jin-Young Kim
66c6a6df191c0295d6cfc8c9
Kyungjune Baek
66c6a6df191c0295d6cfc8ca
Jihwan Kim
66c6a6df191c0295d6cfc8cb
Hyojun Go
claimed_verified
2024-11-20T10:09:53.339Z
630d5aff7dacb93b3359fe26
/avatars/e69c2bcb9d45131b68defae1bdb9d0cc.svg
Hyojun Go 
user
gohyojun15
66c6a6df191c0295d6cfc8cc
Seongsu Ha
66c6a6df191c0295d6cfc8cd
Seokjin Han
66c6a6df191c0295d6cfc8ce
Jiho Jang
66c6a6df191c0295d6cfc8cf
Raehyuk Jung
66c6a6df191c0295d6cfc8d0
Daewoo Kim
66c6a6df191c0295d6cfc8d1
GeunOh Kim
66c6a6df191c0295d6cfc8d2
JongMok Kim
66c6a6df191c0295d6cfc8d3
Jongseok Kim
66c6a6df191c0295d6cfc8d4
Junwan Kim
claimed_verified
2024-12-02T08:52:01.869Z
64c2c45ae818eec6128fdda3
/avatars/d4399e25e6399345e263c7902789047e.svg
Jun-Wan KIM
user
junwann
66c6a6df191c0295d6cfc8d5
Soonwoo Kwon
66c6a6df191c0295d6cfc8d6
Jangwon Lee
66c6a6df191c0295d6cfc8d7
Seungjoon Park
66c6a6df191c0295d6cfc8d8
Minjoon Seo
66c6a6df191c0295d6cfc8d9
Jay Suh
66c6a6df191c0295d6cfc8da
Jaehyuk Yi
66c6a6df191c0295d6cfc8db
Aiden Lee
TWLV-I: Analysis and Insights from Holistic Evaluation on Video
  Foundation Models
In this work, we discuss evaluating video foundation models in a fair and
robust manner. Unlike language or image foundation models, many video
foundation models are evaluated with differing parameters (such as sampling
rate, number of frames, pretraining steps, etc.), making fair and robust
comparisons challenging. Therefore, we present a carefully designed evaluation
framework for measuring two core capabilities of video comprehension:
appearance and motion understanding. Our findings reveal that existing video
foundation models, whether text-supervised like UMT or InternVideo2, or
self-supervised like V-JEPA, exhibit limitations in at least one of these
capabilities. As an alternative, we introduce TWLV-I, a new video foundation
model that constructs robust visual representations for both motion- and
appearance-based videos. Based on the average top-1 accuracy of linear probing
on five action recognition benchmarks, pretrained only on publicly accessible
datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a
7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger
models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a
2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared
to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from
videos of several commonly used video benchmarks, along with evaluation source
code that can directly utilize these embeddings. The code is available on
"https://github.com/twelvelabs-io/video-embeddings-evaluation-framework".
The study introduces TWLV-I, a novel video foundation model that outperforms existing models in video comprehension by constructing robust representations for both motion and appearance.
video foundation models appearance understanding motion understanding linear probing action recognition benchmarks UMT InternVideo2 V-JEPA DFN embedding vectors