A Suite of Generative Tasks for Multi-Level Multimodal Webpage
  Understanding
Webpages have been a rich, scalable resource for vision-language and language
only tasks. Yet only pieces of webpages are kept: image-caption pairs, long
text articles, or raw HTML, never all in one place. Webpage tasks have
resultingly received little attention and structured image-text data left
underused. To study multimodal webpage understanding, we introduce the
Wikipedia Webpage suite (WikiWeb2M) of 2M pages. We verify its utility on three
generative tasks: page description generation, section summarization, and
contextual image captioning. We design a novel attention mechanism Prefix
Global, which selects the most relevant image and text content as global tokens
to attend to the rest of the webpage for context. By using page structure to
separate such tokens, it performs better than full attention with lower
computational complexity. Experiments show that the new annotations from
WikiWeb2M improve task performance compared to data from prior work. We also
include ablations on sequence length, input features, and model size.
64586589aa36a63c42a0cc9c
Andrea Burns
64586589aa36a63c42a0cc9d
Krishna Srinivasan
admin_assigned
2023-05-23T10:19:35.363Z
6266ea037cc13cb44d35a168
/avatars/096d479f403cb561bab1dc13f0fc0339.svg
Krishna Srinivasan
user
krishna2
64586589aa36a63c42a0cc9e
Joshua Ainslie
64586589aa36a63c42a0cc9f
Geoff Brown
64586589aa36a63c42a0cca0
Bryan A. Plummer
64586589aa36a63c42a0cca1
Kate Saenko
64586589aa36a63c42a0cca2
Jianmo Ni
64586589aa36a63c42a0cca3
Mandy Guo
A Suite of Generative Tasks for Multi-Level Multimodal Webpage
  Understanding
Webpages have been a rich, scalable resource for vision-language and language
only tasks. Yet only pieces of webpages are kept: image-caption pairs, long
text articles, or raw HTML, never all in one place. Webpage tasks have
resultingly received little attention and structured image-text data left
underused. To study multimodal webpage understanding, we introduce the
Wikipedia Webpage suite (WikiWeb2M) of 2M pages. We verify its utility on three
generative tasks: page description generation, section summarization, and
contextual image captioning. We design a novel attention mechanism Prefix
Global, which selects the most relevant image and text content as global tokens
to attend to the rest of the webpage for context. By using page structure to
separate such tokens, it performs better than full attention with lower
computational complexity. Experiments show that the new annotations from
WikiWeb2M improve task performance compared to data from prior work. We also
include ablations on sequence length, input features, and model size.
WikiWeb2M, a suite of 2M Wikipedia pages, is introduced to improve multimodal webpage understanding with a novel Prefix Global attention mechanism, enhancing generative tasks like page description, section summarization, and image captioning.
Wikipedia Webpage suite WikiWeb2M multimodal webpage understanding Prefix Global generative tasks page description generation section summarization contextual image captioning global tokens attention mechanism page structure sequence length input features model size