CinePile: A Long Video Question Answering Dataset and Benchmark
<|reference_start|>CinePile: A Long Video Question Answering Dataset and Benchmark: Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be successfully tackled by analyzing just one or a few random frames from a video. To address this issue, we present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. This paper details our innovative approach for creating a question-answer dataset, utilizing advanced LLMs with human-in-the-loop and building upon human-generated raw data. Our comprehensive dataset comprises 305,000 multiple-choice questions (MCQs), covering various visual and multimodal aspects, including temporal comprehension, understanding human-object interactions, and reasoning about events or actions within a scene. Additionally, we evaluate recent video-centric LLMs, both open-source and proprietary, on the test split of our dataset. The findings reveal that even state-of-the-art video-centric LLMs significantly lag behind human performance in these tasks, highlighting the complexity and challenge inherent in video understanding. The dataset is available at https://hf.co/datasets/tomg-group-umd/cinepile<|reference_end|>
@article{rawal2024cinepile:,
    title={CinePile: A Long Video Question Answering Dataset and Benchmark},
    author={Ruchit Rawal, Khalid Saifullah, Miquel Farr'e, Ronen Basri, David
  Jacobs, Gowthami Somepalli, Tom Goldstein},
    journal={arXiv preprint arXiv:2405.08813},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.08813},
    primaryClass={cs.CV cs.LG cs.MM}
}