Video Instruction Tuning With Synthetic Data
<|reference_start|>Video Instruction Tuning With Synthetic Data: The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.<|reference_end|>
@article{zhang2024video,
    title={Video Instruction Tuning With Synthetic Data},
    author={Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu,
  Chunyuan Li},
    journal={arXiv preprint arXiv:2410.02713},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02713},
    primaryClass={cs.CV cs.CL}
}