title: M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for
    Evaluating Foundation Models
summary: Existing benchmarks for evaluating foundation models mainly focus on
  single-document, text-only tasks. However, they often fail to fully capture the
  complexity of research workflows, which typically involve interpreting
  non-textual data and gathering information across multiple documents. To
  address this gap, we introduce M3SciQA, a multi-modal, multi-document
  scientific question answering benchmark designed for a more comprehensive
  evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated
  questions spanning 70 natural language processing paper clusters, where each
  cluster represents a primary paper along with all its cited documents,
  mirroring the workflow of comprehending a single paper by requiring multi-modal
  and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of
  18 foundation models. Our results indicate that current foundation models still
  significantly underperform compared to human experts in multi-modal information
  retrieval and in reasoning across multiple scientific documents. Additionally,
  we explore the implications of these findings for the future advancement of
  applying foundation models in multi-modal scientific literature analysis.
paper_authors._id: 672cd87096a9d93d169922a8
paper_authors.name: Chuhan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:33:09.809Z
paper_authors._id: 65415f1d5168c4f3487a2103
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65415f1d5168c4f3487a2103/qJuzDpOGSDL4E1-L2lGWW.jpeg
paper_authors.fullname: Chuhan Li
paper_authors.type: user
paper_authors.user: ChuhanLi
paper_authors._id: 672cd87096a9d93d169922a9
paper_authors.name: Ziyao Shangguan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:33:04.816Z
paper_authors._id: 65dea56779f827c045b1df96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jtx4JcM5CNgoVyzxTC93S.jpeg
paper_authors.fullname: Ziyao Shangguan
paper_authors.type: user
paper_authors.user: ziyaosg
paper_authors._id: 672cd87096a9d93d169922aa
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:38.117Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 672cd87096a9d93d169922ab
paper_authors.name: Deyuan Li
paper_authors._id: 672cd87096a9d93d169922ac
paper_authors.name: Yixin Liu
paper_authors._id: 672cd87096a9d93d169922ad
paper_authors.name: Arman Cohan
paper_title: M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for
    Evaluating Foundation Models
paper_summary: Existing benchmarks for evaluating foundation models mainly focus on
  single-document, text-only tasks. However, they often fail to fully capture the
  complexity of research workflows, which typically involve interpreting
  non-textual data and gathering information across multiple documents. To
  address this gap, we introduce M3SciQA, a multi-modal, multi-document
  scientific question answering benchmark designed for a more comprehensive
  evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated
  questions spanning 70 natural language processing paper clusters, where each
  cluster represents a primary paper along with all its cited documents,
  mirroring the workflow of comprehending a single paper by requiring multi-modal
  and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of
  18 foundation models. Our results indicate that current foundation models still
  significantly underperform compared to human experts in multi-modal information
  retrieval and in reasoning across multiple scientific documents. Additionally,
  we explore the implications of these findings for the future advancement of
  applying foundation models in multi-modal scientific literature analysis.
paper_ai_keywords: multi-modal multi-document scientific question answering multi-modal, multi-document scientific question answering benchmark natural language processing paper clusters