title: A Suite of Generative Tasks for Multi-Level Multimodal Webpage
    Understanding
summary: Webpages have been a rich, scalable resource for vision-language and language
  only tasks. Yet only pieces of webpages are kept: image-caption pairs, long
  text articles, or raw HTML, never all in one place. Webpage tasks have
  resultingly received little attention and structured image-text data left
  underused. To study multimodal webpage understanding, we introduce the
  Wikipedia Webpage suite (WikiWeb2M) of 2M pages. We verify its utility on three
  generative tasks: page description generation, section summarization, and
  contextual image captioning. We design a novel attention mechanism Prefix
  Global, which selects the most relevant image and text content as global tokens
  to attend to the rest of the webpage for context. By using page structure to
  separate such tokens, it performs better than full attention with lower
  computational complexity. Experiments show that the new annotations from
  WikiWeb2M improve task performance compared to data from prior work. We also
  include ablations on sequence length, input features, and model size.
paper_authors._id: 64586589aa36a63c42a0cc9c
paper_authors.name: Andrea Burns
paper_authors._id: 64586589aa36a63c42a0cc9d
paper_authors.name: Krishna Srinivasan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T10:19:35.363Z
paper_authors._id: 6266ea037cc13cb44d35a168
paper_authors.avatarUrl: /avatars/096d479f403cb561bab1dc13f0fc0339.svg
paper_authors.fullname: Krishna Srinivasan
paper_authors.type: user
paper_authors.user: krishna2
paper_authors._id: 64586589aa36a63c42a0cc9e
paper_authors.name: Joshua Ainslie
paper_authors._id: 64586589aa36a63c42a0cc9f
paper_authors.name: Geoff Brown
paper_authors._id: 64586589aa36a63c42a0cca0
paper_authors.name: Bryan A. Plummer
paper_authors._id: 64586589aa36a63c42a0cca1
paper_authors.name: Kate Saenko
paper_authors._id: 64586589aa36a63c42a0cca2
paper_authors.name: Jianmo Ni
paper_authors._id: 64586589aa36a63c42a0cca3
paper_authors.name: Mandy Guo
paper_title: A Suite of Generative Tasks for Multi-Level Multimodal Webpage
    Understanding
paper_summary: Webpages have been a rich, scalable resource for vision-language and language
  only tasks. Yet only pieces of webpages are kept: image-caption pairs, long
  text articles, or raw HTML, never all in one place. Webpage tasks have
  resultingly received little attention and structured image-text data left
  underused. To study multimodal webpage understanding, we introduce the
  Wikipedia Webpage suite (WikiWeb2M) of 2M pages. We verify its utility on three
  generative tasks: page description generation, section summarization, and
  contextual image captioning. We design a novel attention mechanism Prefix
  Global, which selects the most relevant image and text content as global tokens
  to attend to the rest of the webpage for context. By using page structure to
  separate such tokens, it performs better than full attention with lower
  computational complexity. Experiments show that the new annotations from
  WikiWeb2M improve task performance compared to data from prior work. We also
  include ablations on sequence length, input features, and model size.
paper_ai_summary: WikiWeb2M, a suite of 2M Wikipedia pages, is introduced to improve multimodal webpage understanding with a novel Prefix Global attention mechanism, enhancing generative tasks like page description, section summarization, and image captioning.
paper_ai_keywords: Wikipedia Webpage suite WikiWeb2M multimodal webpage understanding Prefix Global generative tasks page description generation section summarization contextual image captioning global tokens attention mechanism page structure sequence length input features model size