TITLE: Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides 

ABSTRACT: Lecture slide presentations, a sequence of pages that contain text and
figures accompanied by speech, are constructed and presented carefully in order
to optimally transfer knowledge to students. Previous studies in multimedia and
psychology attribute the effectiveness of lecture presentations to their
multimodal nature. As a step toward developing AI to aid in student learning as
intelligent teacher assistants, we introduce the Multimodal Lecture
Presentations dataset as a large-scale benchmark testing the capabilities of
machine learning models in multimodal understanding of educational content. Our
dataset contains aligned slides and spoken language, for 180+ hours of video
and 9000+ slides, with 10 lecturers from various subjects (e.g., computer
science, dentistry, biology). We introduce two research tasks which are
designed as stepping stones towards AI agents that can explain (automatically
captioning a lecture presentation) and illustrate (synthesizing visual figures
to accompany spoken explanations) educational content. We provide manual
annotations to help implement these two research tasks and evaluate
state-of-the-art models on them. Comparing baselines and human student
performances, we find that current models struggle in (1) weak crossmodal
alignment between slides and spoken text, (2) learning novel visual mediums,
(3) technical language, and (4) long-range sequences. Towards addressing this
issue, we also introduce PolyViLT, a multimodal transformer trained with a
multi-instance learning loss that is more effective than current approaches. We
conclude by shedding light on the challenges and opportunities in multimodal
understanding of educational presentations.
Lecture slide presentations, a sequence of pages that contain text and
figures accompanied by speech, are constructed and presented carefully in order
to optimally transfer knowledge to students. Previous studies in multimedia and
psychology attribute the effectiveness of lecture presentations to their
multimodal nature. As a step toward developing AI to aid in student learning as
intelligent teacher assistants, we introduce the Multimodal Lecture
Presentations dataset as a large-scale benchmark testing the capabilities of
machine learning models in multimodal understanding of educational content. Our
dataset contains aligned slides and spoken language, for 180+ hours of video
and 9000+ slides, with 10 lecturers from various subjects (e.g., computer
science, dentistry, biology). We introduce two research tasks which are
designed as stepping stones towards AI agents that can explain (automatically
captioning a lecture presentation) and illustrate (synthesizing visual figures
to accompany spoken explanations) educational content. We provide manual
annotations to help implement these two research tasks and evaluate
state-of-the-art models on them. Comparing baselines and human student
performances, we find that current models struggle in (1) weak crossmodal
alignment between slides and spoken text, (2) learning novel visual mediums,
(3) technical language, and (4) long-range sequences. Towards addressing this
issue, we also introduce PolyViLT, a multimodal transformer trained with a
multi-instance learning loss that is more effective than current approaches. We
conclude by shedding light on the challenges and opportunities in multimodal
understanding of educational presentations.
Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
http://arxiv.org/abs/2208.08080v1