One missing piece in Vision and Language: A Survey on Comics
  Understanding
Vision-language models have recently evolved into versatile systems capable
of high performance across a range of tasks, such as document understanding,
visual question answering, and grounding, often in zero-shot settings. Comics
Understanding, a complex and multifaceted field, stands to greatly benefit from
these advances. Comics, as a medium, combine rich visual and textual
narratives, challenging AI models with tasks that span image classification,
object detection, instance segmentation, and deeper narrative comprehension
through sequential panels. However, the unique structure of comics --
characterized by creative variations in style, reading order, and non-linear
storytelling -- presents a set of challenges distinct from those in other
visual-language domains. In this survey, we present a comprehensive review of
Comics Understanding from both dataset and task perspectives. Our contributions
are fivefold: (1) We analyze the structure of the comics medium, detailing its
distinctive compositional elements; (2) We survey the widely used datasets and
tasks in comics research, emphasizing their role in advancing the field; (3) We
introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy
that redefines vision-language tasks within comics and lays the foundation for
future work; (4) We provide a detailed review and categorization of existing
methods following the LoCU framework; (5) Finally, we highlight current
research challenges and propose directions for future exploration, particularly
in the context of vision-language models applied to comics. This survey is the
first to propose a task-oriented framework for comics intelligence and aims to
guide future research by addressing critical gaps in data availability and task
definition. A project associated with this survey is available at
https://github.com/emanuelevivoli/awesome-comics-understanding.
66e90e49955eeec85e4f6842
Emanuele Vivoli
admin_assigned
2024-09-17T09:30:07.682Z
6065a9cbe43e52694178ed78
https://cdn-avatars.huggingface.co/v1/production/uploads/6065a9cbe43e52694178ed78/TwnrOfTacrwtLnBC8cS8e.jpeg
Emanuele Vivoli
user
emanuelevivoli
66e90e49955eeec85e4f6843
Andrey Barsky
66e90e49955eeec85e4f6844
Mohamed Ali Souibgui
claimed_verified
2024-09-17T15:08:12.305Z
66e98dcf899bdb384bd951d6
https://cdn-avatars.huggingface.co/v1/production/uploads/66e98dcf899bdb384bd951d6/6nCMkogUuQWJm3FR05Ygc.jpeg
Mohamed Ali Souibgui
user
msouibgui
66e90e49955eeec85e4f6845
Artemis LLabres
claimed_verified
2024-09-17T11:31:49.342Z
630f3e904dbbbb6f667eceb7
https://cdn-avatars.huggingface.co/v1/production/uploads/630f3e904dbbbb6f667eceb7/Zyzby4Ruo70wmNhmYC_v4.jpeg
Artemis Llabrés
user
Llabres
66e90e49955eeec85e4f6846
Marco Bertini
admin_assigned
2024-09-17T09:31:38.153Z
6319e9b78b8425c8fe653dd6
/avatars/b1abd8bc31e7e39d7d563cc275656e6c.svg
Marco Bertini
user
MarcoBertini
66e90e49955eeec85e4f6847
Dimosthenis Karatzas
One missing piece in Vision and Language: A Survey on Comics
  Understanding
Vision-language models have recently evolved into versatile systems capable
of high performance across a range of tasks, such as document understanding,
visual question answering, and grounding, often in zero-shot settings. Comics
Understanding, a complex and multifaceted field, stands to greatly benefit from
these advances. Comics, as a medium, combine rich visual and textual
narratives, challenging AI models with tasks that span image classification,
object detection, instance segmentation, and deeper narrative comprehension
through sequential panels. However, the unique structure of comics --
characterized by creative variations in style, reading order, and non-linear
storytelling -- presents a set of challenges distinct from those in other
visual-language domains. In this survey, we present a comprehensive review of
Comics Understanding from both dataset and task perspectives. Our contributions
are fivefold: (1) We analyze the structure of the comics medium, detailing its
distinctive compositional elements; (2) We survey the widely used datasets and
tasks in comics research, emphasizing their role in advancing the field; (3) We
introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy
that redefines vision-language tasks within comics and lays the foundation for
future work; (4) We provide a detailed review and categorization of existing
methods following the LoCU framework; (5) Finally, we highlight current
research challenges and propose directions for future exploration, particularly
in the context of vision-language models applied to comics. This survey is the
first to propose a task-oriented framework for comics intelligence and aims to
guide future research by addressing critical gaps in data availability and task
definition. A project associated with this survey is available at
https://github.com/emanuelevivoli/awesome-comics-understanding.
A comprehensive survey addresses the challenges of comics understanding, introducing a new taxonomy and reviewing methods within the context of vision-language models.
vision-language models document understanding visual question answering grounding Comics Understanding image classification object detection instance segmentation narrative comprehension sequential panels Layer of Comics Understanding (LoCU) framework