title: LongVideoBench: A Benchmark for Long-context Interleaved Video-Language
    Understanding
summary: Large multimodal models (LMMs) are processing increasingly longer and richer
  inputs. Albeit the progress, few public benchmark is available to measure such
  development. To mitigate this gap, we introduce LongVideoBench, a
  question-answering benchmark that features video-language interleaved inputs up
  to an hour long. Our benchmark includes 3,763 varying-length web-collected
  videos with their subtitles across diverse themes, designed to comprehensively
  evaluate LMMs on long-term multimodal understanding. To achieve this, we
  interpret the primary challenge as to accurately retrieve and reason over
  detailed multimodal information from long inputs. As such, we formulate a novel
  video question-answering task termed referring reasoning. Specifically, as part
  of the question, it contains a referring query that references related video
  contexts, called referred context. The model is then required to reason over
  relevant video details from the referred context. Following the paradigm of
  referring reasoning, we curate 6,678 human-annotated multiple-choice questions
  in 17 fine-grained categories, establishing one of the most comprehensive
  benchmarks for long-form video understanding. Evaluations suggest that the
  LongVideoBench presents significant challenges even for the most advanced
  proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their
  open-source counterparts show an even larger performance gap. In addition, our
  results indicate that model performance on the benchmark improves only when
  they are capable of processing more frames, positioning LongVideoBench as a
  valuable benchmark for evaluating future-generation long-context LMMs.
paper_authors._id: 669f43ba04a09709a6b378f8
paper_authors.name: Haoning Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:47:21.755Z
paper_authors._id: 63047ed2412a1b9d381b09c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
paper_authors.fullname: Haoning Wu, Teo
paper_authors.type: user
paper_authors.user: teowu
paper_authors._id: 669f43ba04a09709a6b378f9
paper_authors.name: Dongxu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:22:56.086Z
paper_authors._id: 6220388b354ac333eb998319
paper_authors.avatarUrl: /avatars/e04ebeb57f9dc9aa2f6d4732737f6cb8.svg
paper_authors.fullname: Dongxu Li
paper_authors.type: user
paper_authors.user: dxli
paper_authors._id: 669f43ba04a09709a6b378fa
paper_authors.name: Bei Chen
paper_authors._id: 669f43ba04a09709a6b378fb
paper_authors.name: Junnan Li
paper_title: LongVideoBench: A Benchmark for Long-context Interleaved Video-Language
    Understanding
paper_summary: Large multimodal models (LMMs) are processing increasingly longer and richer
  inputs. Albeit the progress, few public benchmark is available to measure such
  development. To mitigate this gap, we introduce LongVideoBench, a
  question-answering benchmark that features video-language interleaved inputs up
  to an hour long. Our benchmark includes 3,763 varying-length web-collected
  videos with their subtitles across diverse themes, designed to comprehensively
  evaluate LMMs on long-term multimodal understanding. To achieve this, we
  interpret the primary challenge as to accurately retrieve and reason over
  detailed multimodal information from long inputs. As such, we formulate a novel
  video question-answering task termed referring reasoning. Specifically, as part
  of the question, it contains a referring query that references related video
  contexts, called referred context. The model is then required to reason over
  relevant video details from the referred context. Following the paradigm of
  referring reasoning, we curate 6,678 human-annotated multiple-choice questions
  in 17 fine-grained categories, establishing one of the most comprehensive
  benchmarks for long-form video understanding. Evaluations suggest that the
  LongVideoBench presents significant challenges even for the most advanced
  proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their
  open-source counterparts show an even larger performance gap. In addition, our
  results indicate that model performance on the benchmark improves only when
  they are capable of processing more frames, positioning LongVideoBench as a
  valuable benchmark for evaluating future-generation long-context LMMs.
paper_ai_summary: LongVideoBench is a question-answering benchmark featuring interleaved video-language inputs up to an hour long, designed to evaluate long-term multimodal understanding and referring reasoning.
paper_ai_keywords: multimodal models multimodal understanding referring reasoning multiple-choice questions fine-grained categories long-form video understanding frames