title: Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and
    Language Models
summary: Conversation agents fueled by Large Language Models (LLMs) are providing a
  new way to interact with visual data. While there have been initial attempts
  for image-based conversation models, this work addresses the underexplored
  field of video-based conversation by introducing Video-ChatGPT. It is a
  multimodal model that merges a video-adapted visual encoder with a LLM. The
  model is capable of understanding and generating human-like conversations about
  videos. We introduce a new dataset of 100,000 video-instruction pairs used to
  train Video-ChatGPT acquired via manual and semi-automated pipeline that is
  easily scalable and robust to label noise. We also develop a quantiative
  evaluation framework for video-based dialogue models to objectively analyse the
  strengths and weaknesses of proposed models. Our code, models, instruction-sets
  and demo are released at https://github.com/mbzuai-oryx/Video-ChatGPT.
paper_authors._id: 64828c62e4bbb1c2dd322d1e
paper_authors.name: Muhammad Maaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T09:12:33.798Z
paper_authors._id: 64807585856901b0edb8d68b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
paper_authors.fullname: Muhammad Maaz
paper_authors.type: user
paper_authors.user: mmaaz60
paper_authors._id: 64828c62e4bbb1c2dd322d1f
paper_authors.name: Hanoona Rasheed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:39:27.933Z
paper_authors._id: 64636b2551fa6e6306046293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
paper_authors.fullname: Hanoona Rasheed
paper_authors.type: user
paper_authors.user: Hanoona
paper_authors._id: 64828c62e4bbb1c2dd322d20
paper_authors.name: Salman Khan
paper_authors._id: 64828c62e4bbb1c2dd322d21
paper_authors.name: Fahad Shahbaz Khan
paper_title: Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and
    Language Models
paper_summary: Conversation agents fueled by Large Language Models (LLMs) are providing a
  new way to interact with visual data. While there have been initial attempts
  for image-based conversation models, this work addresses the underexplored
  field of video-based conversation by introducing Video-ChatGPT. It is a
  multimodal model that merges a video-adapted visual encoder with a LLM. The
  model is capable of understanding and generating human-like conversations about
  videos. We introduce a new dataset of 100,000 video-instruction pairs used to
  train Video-ChatGPT acquired via manual and semi-automated pipeline that is
  easily scalable and robust to label noise. We also develop a quantiative
  evaluation framework for video-based dialogue models to objectively analyse the
  strengths and weaknesses of proposed models. Our code, models, instruction-sets
  and demo are released at https://github.com/mbzuai-oryx/Video-ChatGPT.
paper_ai_summary: Video-ChatGPT is a multimodal model combining a video-adapted visual encoder with a Large Language Model to understand and generate human-like conversations about videos, trained on a scalable dataset of video-instruction pairs.
paper_ai_keywords: Large Language Models multimodal model video-adapted visual encoder human-like conversations video-based conversation video-instruction pairs manual and semi-automated pipeline quantiative evaluation framework video-based dialogue models