title: SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
summary: Seeking answers to questions within long scientific research articles is a
  crucial area of study that aids readers in quickly addressing their inquiries.
  However, existing question-answering (QA) datasets based on scientific papers
  are limited in scale and focus solely on textual content. To address this
  limitation, we introduce SPIQA (Scientific Paper Image Question Answering), the
  first large-scale QA dataset specifically designed to interpret complex figures
  and tables within the context of scientific research articles across various
  domains of computer science. Leveraging the breadth of expertise and ability of
  multimodal large language models (MLLMs) to understand figures, we employ
  automatic and manual curation to create the dataset. We craft an
  information-seeking task involving multiple images that cover a wide variety of
  plots, charts, tables, schematic diagrams, and result visualizations. SPIQA
  comprises 270K questions divided into training, validation, and three different
  evaluation splits. Through extensive experiments with 12 prominent foundational
  models, we evaluate the ability of current multimodal systems to comprehend the
  nuanced aspects of research articles. Additionally, we propose a
  Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that
  allows fine-grained, step-by-step assessment and improves model performance. We
  further explore the upper bounds of performance enhancement with additional
  textual information, highlighting its promising potential for future research
  and the dataset's impact on revolutionizing how we interact with scientific
  literature.
paper_authors._id: 6694e325c5f4e75ee510029d
paper_authors.name: Shraman Pramanick
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:54.292Z
paper_authors._id: 64aec10a9135f79b09117476
paper_authors.avatarUrl: /avatars/88a6bcfeee99c12496940575a2fc65e9.svg
paper_authors.fullname: Shraman Pramanick
paper_authors.type: user
paper_authors.user: Shraman
paper_authors._id: 6694e325c5f4e75ee510029e
paper_authors.name: Rama Chellappa
paper_authors._id: 6694e325c5f4e75ee510029f
paper_authors.name: Subhashini Venugopalan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-20T13:16:06.318Z
paper_authors._id: 646125a5933afb0106a9dabb
paper_authors.avatarUrl: /avatars/ccef359e4442f560a87fd66228e54c8b.svg
paper_authors.fullname: Subhashini
paper_authors.type: user
paper_authors.user: vsubhashini
paper_title: SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
paper_summary: Seeking answers to questions within long scientific research articles is a
  crucial area of study that aids readers in quickly addressing their inquiries.
  However, existing question-answering (QA) datasets based on scientific papers
  are limited in scale and focus solely on textual content. To address this
  limitation, we introduce SPIQA (Scientific Paper Image Question Answering), the
  first large-scale QA dataset specifically designed to interpret complex figures
  and tables within the context of scientific research articles across various
  domains of computer science. Leveraging the breadth of expertise and ability of
  multimodal large language models (MLLMs) to understand figures, we employ
  automatic and manual curation to create the dataset. We craft an
  information-seeking task involving multiple images that cover a wide variety of
  plots, charts, tables, schematic diagrams, and result visualizations. SPIQA
  comprises 270K questions divided into training, validation, and three different
  evaluation splits. Through extensive experiments with 12 prominent foundational
  models, we evaluate the ability of current multimodal systems to comprehend the
  nuanced aspects of research articles. Additionally, we propose a
  Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that
  allows fine-grained, step-by-step assessment and improves model performance. We
  further explore the upper bounds of performance enhancement with additional
  textual information, highlighting its promising potential for future research
  and the dataset's impact on revolutionizing how we interact with scientific
  literature.
paper_ai_summary: SPIQA addresses the limitations of existing QA datasets by introducing a large-scale dataset for interpreting complex figures and tables in scientific papers, using multimodal large language models and a Chain-of-Thought evaluation strategy.
paper_ai_keywords: multimodal large language models Chain-of-Thought (CoT) evaluation strategy