Video-STaR: Self-Training Enables Video Instruction Tuning with Any
  Supervision
The performance of Large Vision Language Models (LVLMs) is dependent on the
size and quality of their training datasets. Existing video instruction tuning
datasets lack diversity as they are derived by prompting large language models
with video captions to generate question-answer pairs, and are therefore mostly
descriptive. Meanwhile, many labeled video datasets with diverse labels and
supervision exist - however, we find that their integration into LVLMs is
non-trivial. Herein, we present Video Self-Training with augmented Reasoning
(Video-STaR), the first video self-training approach. Video-STaR allows the
utilization of any labeled video dataset for video instruction tuning. In
Video-STaR, an LVLM cycles between instruction generation and finetuning, which
we show (I) improves general video understanding and (II) adapts LVLMs to novel
downstream tasks with existing supervision. During generation, an LVLM is
prompted to propose an answer. The answers are then filtered only to those that
contain the original video labels, and the LVLM is then re-trained on the
generated dataset. By only training on generated answers that contain the
correct video labels, Video-STaR utilizes these existing video labels as weak
supervision for video instruction tuning. Our results demonstrate that
Video-STaR-enhanced LVLMs exhibit improved performance in (I) general video QA,
where TempCompass performance improved by 10%, and (II) on downstream tasks,
where Video-STaR improved Kinetics700-QA accuracy by 20% and action quality
assessment on FineDiving by 15%.
668cb636d10c3be5d3576f8f
Orr Zohar
claimed_verified
2024-07-09T07:42:01.105Z
648c9605565e3a44f3c9bb7b
https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
Orr Zohar
user
orrzohar
668cb636d10c3be5d3576f90
Xiaohan Wang
admin_assigned
2024-07-10T10:34:31.848Z
65703fab7f50602340d23704
/avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
Xiaohan Wang
user
nicholswang
668cb636d10c3be5d3576f91
Yonatan Bitton
admin_assigned
2024-07-10T10:34:39.511Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
668cb636d10c3be5d3576f92
Idan Szpektor
668cb636d10c3be5d3576f93
Serena Yeung-Levy
Video-STaR: Self-Training Enables Video Instruction Tuning with Any
  Supervision
The performance of Large Vision Language Models (LVLMs) is dependent on the
size and quality of their training datasets. Existing video instruction tuning
datasets lack diversity as they are derived by prompting large language models
with video captions to generate question-answer pairs, and are therefore mostly
descriptive. Meanwhile, many labeled video datasets with diverse labels and
supervision exist - however, we find that their integration into LVLMs is
non-trivial. Herein, we present Video Self-Training with augmented Reasoning
(Video-STaR), the first video self-training approach. Video-STaR allows the
utilization of any labeled video dataset for video instruction tuning. In
Video-STaR, an LVLM cycles between instruction generation and finetuning, which
we show (I) improves general video understanding and (II) adapts LVLMs to novel
downstream tasks with existing supervision. During generation, an LVLM is
prompted to propose an answer. The answers are then filtered only to those that
contain the original video labels, and the LVLM is then re-trained on the
generated dataset. By only training on generated answers that contain the
correct video labels, Video-STaR utilizes these existing video labels as weak
supervision for video instruction tuning. Our results demonstrate that
Video-STaR-enhanced LVLMs exhibit improved performance in (I) general video QA,
where TempCompass performance improved by 10%, and (II) on downstream tasks,
where Video-STaR improved Kinetics700-QA accuracy by 20% and action quality
assessment on FineDiving by 15%.
Video Self-Training with augmented Reasoning (Video-STaR) enhances Large Vision Language Models by leveraging labeled video datasets for video instruction tuning, improving general video understanding and downstream task performance.
Large Vision Language Models Video Self-Training Video-STaR video instruction tuning instruction generation weak supervision video QA Kinetics700-QA FineDiving