VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference
  Optimization for Large Video Models
Large Video Models (LVMs) built upon Large Language Models (LLMs) have shown
promise in video understanding but often suffer from misalignment with human
intuition and video hallucination issues. To address these challenges, we
introduce VistaDPO, a novel framework for Video Hierarchical Spatial-Temporal
Direct Preference Optimization. VistaDPO enhances text-video preference
alignment across three hierarchical levels: i) Instance Level, aligning overall
video content with responses; ii) Temporal Level, aligning video temporal
semantics with event descriptions; and iii) Perceptive Level, aligning spatial
objects with language tokens. Given the lack of datasets for fine-grained
video-language preference alignment, we construct VistaDPO-7k, a dataset of
7.2K QA pairs annotated with chosen and rejected responses, along with
spatial-temporal grounding information such as timestamps, keyframes, and
bounding boxes. Extensive experiments on benchmarks such as Video
Hallucination, Video QA, and Captioning performance tasks demonstrate that
VistaDPO significantly improves the performance of existing LVMs, effectively
mitigating video-language misalignment and hallucination. The code and data are
available at https://github.com/HaroldChen19/VistaDPO.
6801b62608d748addc187952
Haojian Huang
6801b62608d748addc187953
Haodong Chen
claimed_verified
2025-04-18T09:39:11.374Z
6570450a78d7aca0c361a177
https://cdn-avatars.huggingface.co/v1/production/uploads/6570450a78d7aca0c361a177/z0GrnXEsjK2_G-hFfQhKv.jpeg
Harold Chen
user
Harold328
6801b62608d748addc187954
Shengqiong Wu
claimed_verified
2025-04-18T09:39:08.857Z
64c139d867eff857ea51caa8
/avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
Shengqiong Wu
user
ChocoWu
6801b62608d748addc187955
Meng Luo
claimed_verified
2025-04-22T10:13:20.029Z
64ad1c0bad6218d51a07b54e
/avatars/0f84d9a51c6ca9bcef44de2d7c707d9b.svg
LUO MENG
user
Eureka-Leo
6801b62608d748addc187956
Jinlan Fu
6801b62608d748addc187957
Xinya Du
6801b62608d748addc187958
Hanwang Zhang
6801b62608d748addc187959
Hao Fei
claimed_verified
2025-04-18T09:39:06.521Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
user
scofield7419
VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference
  Optimization for Large Video Models
Large Video Models (LVMs) built upon Large Language Models (LLMs) have shown
promise in video understanding but often suffer from misalignment with human
intuition and video hallucination issues. To address these challenges, we
introduce VistaDPO, a novel framework for Video Hierarchical Spatial-Temporal
Direct Preference Optimization. VistaDPO enhances text-video preference
alignment across three hierarchical levels: i) Instance Level, aligning overall
video content with responses; ii) Temporal Level, aligning video temporal
semantics with event descriptions; and iii) Perceptive Level, aligning spatial
objects with language tokens. Given the lack of datasets for fine-grained
video-language preference alignment, we construct VistaDPO-7k, a dataset of
7.2K QA pairs annotated with chosen and rejected responses, along with
spatial-temporal grounding information such as timestamps, keyframes, and
bounding boxes. Extensive experiments on benchmarks such as Video
Hallucination, Video QA, and Captioning performance tasks demonstrate that
VistaDPO significantly improves the performance of existing LVMs, effectively
mitigating video-language misalignment and hallucination. The code and data are
available at https://github.com/HaroldChen19/VistaDPO.
VistaDPO, a new framework, enhances text-video preference alignment through hierarchical spatial-temporal optimization, addressing misalignment and hallucination in large video models.
Large Video Models Large Language Models Video Hierarchical Spatial-Temporal Direct Preference Optimization Instance Level Temporal Level Perceptive Level VistaDPO-7k video hallucination Video QA Captioning