SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
Seeking answers to questions within long scientific research articles is a
crucial area of study that aids readers in quickly addressing their inquiries.
However, existing question-answering (QA) datasets based on scientific papers
are limited in scale and focus solely on textual content. To address this
limitation, we introduce SPIQA (Scientific Paper Image Question Answering), the
first large-scale QA dataset specifically designed to interpret complex figures
and tables within the context of scientific research articles across various
domains of computer science. Leveraging the breadth of expertise and ability of
multimodal large language models (MLLMs) to understand figures, we employ
automatic and manual curation to create the dataset. We craft an
information-seeking task involving multiple images that cover a wide variety of
plots, charts, tables, schematic diagrams, and result visualizations. SPIQA
comprises 270K questions divided into training, validation, and three different
evaluation splits. Through extensive experiments with 12 prominent foundational
models, we evaluate the ability of current multimodal systems to comprehend the
nuanced aspects of research articles. Additionally, we propose a
Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that
allows fine-grained, step-by-step assessment and improves model performance. We
further explore the upper bounds of performance enhancement with additional
textual information, highlighting its promising potential for future research
and the dataset's impact on revolutionizing how we interact with scientific
literature.
6694e325c5f4e75ee510029d
Shraman Pramanick
claimed_verified
2024-07-22T07:10:54.292Z
64aec10a9135f79b09117476
/avatars/83db978a906782b6df71d178b7a0df89.svg
Shraman Pramanick
user
Shraman
6694e325c5f4e75ee510029e
Rama Chellappa
6694e325c5f4e75ee510029f
Subhashini Venugopalan
extracted_confirmed
2025-03-20T13:16:06.318Z
646125a5933afb0106a9dabb
/avatars/ccef359e4442f560a87fd66228e54c8b.svg
Subhashini
user
vsubhashini
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
Seeking answers to questions within long scientific research articles is a
crucial area of study that aids readers in quickly addressing their inquiries.
However, existing question-answering (QA) datasets based on scientific papers
are limited in scale and focus solely on textual content. To address this
limitation, we introduce SPIQA (Scientific Paper Image Question Answering), the
first large-scale QA dataset specifically designed to interpret complex figures
and tables within the context of scientific research articles across various
domains of computer science. Leveraging the breadth of expertise and ability of
multimodal large language models (MLLMs) to understand figures, we employ
automatic and manual curation to create the dataset. We craft an
information-seeking task involving multiple images that cover a wide variety of
plots, charts, tables, schematic diagrams, and result visualizations. SPIQA
comprises 270K questions divided into training, validation, and three different
evaluation splits. Through extensive experiments with 12 prominent foundational
models, we evaluate the ability of current multimodal systems to comprehend the
nuanced aspects of research articles. Additionally, we propose a
Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that
allows fine-grained, step-by-step assessment and improves model performance. We
further explore the upper bounds of performance enhancement with additional
textual information, highlighting its promising potential for future research
and the dataset's impact on revolutionizing how we interact with scientific
literature.
SPIQA addresses the limitations of existing QA datasets by introducing a large-scale dataset for interpreting complex figures and tables in scientific papers, using multimodal large language models and a Chain-of-Thought evaluation strategy.
multimodal large language models Chain-of-Thought (CoT) evaluation strategy