title: Video-STaR: Self-Training Enables Video Instruction Tuning with Any
    Supervision
summary: The performance of Large Vision Language Models (LVLMs) is dependent on the
  size and quality of their training datasets. Existing video instruction tuning
  datasets lack diversity as they are derived by prompting large language models
  with video captions to generate question-answer pairs, and are therefore mostly
  descriptive. Meanwhile, many labeled video datasets with diverse labels and
  supervision exist - however, we find that their integration into LVLMs is
  non-trivial. Herein, we present Video Self-Training with augmented Reasoning
  (Video-STaR), the first video self-training approach. Video-STaR allows the
  utilization of any labeled video dataset for video instruction tuning. In
  Video-STaR, an LVLM cycles between instruction generation and finetuning, which
  we show (I) improves general video understanding and (II) adapts LVLMs to novel
  downstream tasks with existing supervision. During generation, an LVLM is
  prompted to propose an answer. The answers are then filtered only to those that
  contain the original video labels, and the LVLM is then re-trained on the
  generated dataset. By only training on generated answers that contain the
  correct video labels, Video-STaR utilizes these existing video labels as weak
  supervision for video instruction tuning. Our results demonstrate that
  Video-STaR-enhanced LVLMs exhibit improved performance in (I) general video QA,
  where TempCompass performance improved by 10%, and (II) on downstream tasks,
  where Video-STaR improved Kinetics700-QA accuracy by 20% and action quality
  assessment on FineDiving by 15%.
paper_authors._id: 668cb636d10c3be5d3576f8f
paper_authors.name: Orr Zohar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:42:01.105Z
paper_authors._id: 648c9605565e3a44f3c9bb7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
paper_authors.fullname: Orr Zohar
paper_authors.type: user
paper_authors.user: orrzohar
paper_authors._id: 668cb636d10c3be5d3576f90
paper_authors.name: Xiaohan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:34:31.848Z
paper_authors._id: 65703fab7f50602340d23704
paper_authors.avatarUrl: /avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
paper_authors.fullname: Xiaohan Wang
paper_authors.type: user
paper_authors.user: nicholswang
paper_authors._id: 668cb636d10c3be5d3576f91
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:34:39.511Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 668cb636d10c3be5d3576f92
paper_authors.name: Idan Szpektor
paper_authors._id: 668cb636d10c3be5d3576f93
paper_authors.name: Serena Yeung-Levy
paper_title: Video-STaR: Self-Training Enables Video Instruction Tuning with Any
    Supervision
paper_summary: The performance of Large Vision Language Models (LVLMs) is dependent on the
  size and quality of their training datasets. Existing video instruction tuning
  datasets lack diversity as they are derived by prompting large language models
  with video captions to generate question-answer pairs, and are therefore mostly
  descriptive. Meanwhile, many labeled video datasets with diverse labels and
  supervision exist - however, we find that their integration into LVLMs is
  non-trivial. Herein, we present Video Self-Training with augmented Reasoning
  (Video-STaR), the first video self-training approach. Video-STaR allows the
  utilization of any labeled video dataset for video instruction tuning. In
  Video-STaR, an LVLM cycles between instruction generation and finetuning, which
  we show (I) improves general video understanding and (II) adapts LVLMs to novel
  downstream tasks with existing supervision. During generation, an LVLM is
  prompted to propose an answer. The answers are then filtered only to those that
  contain the original video labels, and the LVLM is then re-trained on the
  generated dataset. By only training on generated answers that contain the
  correct video labels, Video-STaR utilizes these existing video labels as weak
  supervision for video instruction tuning. Our results demonstrate that
  Video-STaR-enhanced LVLMs exhibit improved performance in (I) general video QA,
  where TempCompass performance improved by 10%, and (II) on downstream tasks,
  where Video-STaR improved Kinetics700-QA accuracy by 20% and action quality
  assessment on FineDiving by 15%.
paper_ai_summary: Video Self-Training with augmented Reasoning (Video-STaR) enhances Large Vision Language Models by leveraging labeled video datasets for video instruction tuning, improving general video understanding and downstream task performance.
paper_ai_keywords: Large Vision Language Models Video Self-Training Video-STaR video instruction tuning instruction generation weak supervision video QA Kinetics700-QA FineDiving