title: Video Instruction Tuning With Synthetic Data
summary: The development of video large multimodal models (LMMs) has been hindered by
  the difficulty of curating large amounts of high-quality raw data from the web.
  To address this, we propose an alternative approach by creating a high-quality
  synthetic dataset specifically for video instruction-following, namely
  LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
  open-ended question-answering (QA), and multiple-choice QA. By training on this
  dataset, in combination with existing visual instruction tuning data, we
  introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
  LLaVA-Video achieves strong performance across various video benchmarks,
  highlighting the effectiveness of our dataset. We plan to release the dataset,
  its generation pipeline, and the model checkpoints.
paper_authors._id: 66ff4f5e5a359c1af1169664
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:39:48.736Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 66ff4f5e5a359c1af1169665
paper_authors.name: Jinming Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:40:01.301Z
paper_authors._id: 652fbe8cb2acab0b82f855a6
paper_authors.avatarUrl: /avatars/c35672b2229bb4f986ef01c61211c3f2.svg
paper_authors.fullname: Jinming Wu
paper_authors.type: user
paper_authors.user: kimingng
paper_authors._id: 66ff4f5e5a359c1af1169666
paper_authors.name: Wei Li
paper_authors._id: 66ff4f5e5a359c1af1169667
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:56.361Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 66ff4f5e5a359c1af1169668
paper_authors.name: Zejun Ma
paper_authors._id: 66ff4f5e5a359c1af1169669
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:40:14.432Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 66ff4f5e5a359c1af116966a
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:40:08.501Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_title: Video Instruction Tuning With Synthetic Data
paper_summary: The development of video large multimodal models (LMMs) has been hindered by
  the difficulty of curating large amounts of high-quality raw data from the web.
  To address this, we propose an alternative approach by creating a high-quality
  synthetic dataset specifically for video instruction-following, namely
  LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
  open-ended question-answering (QA), and multiple-choice QA. By training on this
  dataset, in combination with existing visual instruction tuning data, we
  introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
  LLaVA-Video achieves strong performance across various video benchmarks,
  highlighting the effectiveness of our dataset. We plan to release the dataset,
  its generation pipeline, and the model checkpoints.
paper_ai_keywords: video large multimodal models (LMMs) synthetic dataset video instruction-following LLaVA-Video-178K detailed captioning open-ended question-answering (QA) multiple-choice QA visual instruction tuning LLaVA-Video video benchmarks