PosterSum: A Multimodal Benchmark for Scientific Poster Summarization
Generating accurate and concise textual summaries from multimodal documents
is challenging, especially when dealing with visually complex content like
scientific posters. We introduce PosterSum, a novel benchmark to advance the
development of vision-language models that can understand and summarize
scientific posters into research paper abstracts. Our dataset contains 16,305
conference posters paired with their corresponding abstracts as summaries. Each
poster is provided in image format and presents diverse visual understanding
challenges, such as complex layouts, dense text regions, tables, and figures.
We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on
PosterSum and demonstrate that they struggle to accurately interpret and
summarize scientific posters. We propose Segment & Summarize, a hierarchical
method that outperforms current MLLMs on automated metrics, achieving a 3.14%
gain in ROUGE-L. This will serve as a starting point for future research on
poster summarization.
67bff9608d761fc6a75e24ad
Rohit Saxena
claimed_verified
2025-02-27T09:13:54.284Z
657ccbf2869d5bb0e53b482f
/avatars/2eae5a10bdc14814a04d9f255f16de6b.svg
Rohit Saxena
user
rohitsaxena
67bff9608d761fc6a75e24ae
Pasquale Minervini
67bff9608d761fc6a75e24af
Frank Keller
PosterSum: A Multimodal Benchmark for Scientific Poster Summarization
Generating accurate and concise textual summaries from multimodal documents
is challenging, especially when dealing with visually complex content like
scientific posters. We introduce PosterSum, a novel benchmark to advance the
development of vision-language models that can understand and summarize
scientific posters into research paper abstracts. Our dataset contains 16,305
conference posters paired with their corresponding abstracts as summaries. Each
poster is provided in image format and presents diverse visual understanding
challenges, such as complex layouts, dense text regions, tables, and figures.
We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on
PosterSum and demonstrate that they struggle to accurately interpret and
summarize scientific posters. We propose Segment & Summarize, a hierarchical
method that outperforms current MLLMs on automated metrics, achieving a 3.14%
gain in ROUGE-L. This will serve as a starting point for future research on
poster summarization.
A benchmark named PosterSum is introduced to evaluate and improve vision-language models' ability to summarize scientific posters into research paper abstracts, demonstrating that hierarchical methods like Segment & Summarize outperform current models.
Multimodal Large Language Models MLLMs Segment & Summarize ROUGE-L