title: FormNetV2: Multimodal Graph Contrastive Learning for Form Document
    Information Extraction
summary: The recent advent of self-supervised pre-training techniques has led to a
  surge in the use of multimodal learning in form document understanding.
  However, existing approaches that extend the mask language modeling to other
  modalities require careful multi-task tuning, complex reconstruction target
  designs, or additional pre-training data. In FormNetV2, we introduce a
  centralized multimodal graph contrastive learning strategy to unify
  self-supervised pre-training for all modalities in one loss. The graph
  contrastive objective maximizes the agreement of multimodal representations,
  providing a natural interplay for all modalities without special customization.
  In addition, we extract image features within the bounding box that joins a
  pair of tokens connected by a graph edge, capturing more targeted visual cues
  without loading a sophisticated and separately pre-trained image embedder.
  FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
  and Payment benchmarks with a more compact model size.
paper_authors._id: 6454624706728ff79a382477
paper_authors.name: Chen-Yu Lee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-08T08:38:20.262Z
paper_authors._id: 64d1fe784dfd5df7076bfd1a
paper_authors.avatarUrl: /avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
paper_authors.fullname: Chen-Yu Lee
paper_authors.type: user
paper_authors.user: chenyulee
paper_authors._id: 6454624706728ff79a382478
paper_authors.name: Chun-Liang Li
paper_authors._id: 6454624706728ff79a382479
paper_authors.name: Hao Zhang
paper_authors._id: 6454624706728ff79a38247a
paper_authors.name: Timothy Dozat
paper_authors._id: 6454624706728ff79a38247b
paper_authors.name: Vincent Perot
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T09:38:20.186Z
paper_authors._id: 64e3f501bbee0c24b95e20c2
paper_authors.avatarUrl: /avatars/bae7b5297e779d91da21ff50389dd90a.svg
paper_authors.fullname: Vincent
paper_authors.type: user
paper_authors.user: vperot
paper_authors._id: 6454624706728ff79a38247c
paper_authors.name: Guolong Su
paper_authors._id: 6454624706728ff79a38247d
paper_authors.name: Xiang Zhang
paper_authors._id: 6454624706728ff79a38247e
paper_authors.name: Kihyuk Sohn
paper_authors._id: 6454624706728ff79a38247f
paper_authors.name: Nikolai Glushnev
paper_authors._id: 6454624706728ff79a382480
paper_authors.name: Renshen Wang
paper_authors._id: 6454624706728ff79a382481
paper_authors.name: Joshua Ainslie
paper_authors._id: 6454624706728ff79a382482
paper_authors.name: Shangbang Long
paper_authors._id: 6454624706728ff79a382483
paper_authors.name: Siyang Qin
paper_authors._id: 6454624706728ff79a382484
paper_authors.name: Yasuhisa Fujii
paper_authors._id: 6454624706728ff79a382485
paper_authors.name: Nan Hua
paper_authors._id: 6454624706728ff79a382486
paper_authors.name: Tomas Pfister
paper_title: FormNetV2: Multimodal Graph Contrastive Learning for Form Document
    Information Extraction
paper_summary: The recent advent of self-supervised pre-training techniques has led to a
  surge in the use of multimodal learning in form document understanding.
  However, existing approaches that extend the mask language modeling to other
  modalities require careful multi-task tuning, complex reconstruction target
  designs, or additional pre-training data. In FormNetV2, we introduce a
  centralized multimodal graph contrastive learning strategy to unify
  self-supervised pre-training for all modalities in one loss. The graph
  contrastive objective maximizes the agreement of multimodal representations,
  providing a natural interplay for all modalities without special customization.
  In addition, we extract image features within the bounding box that joins a
  pair of tokens connected by a graph edge, capturing more targeted visual cues
  without loading a sophisticated and separately pre-trained image embedder.
  FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
  and Payment benchmarks with a more compact model size.
paper_ai_summary: FormNetV2 uses a centralized multimodal graph contrastive learning approach to improve self-supervised pre-training and achieves state-of-the-art performance on form understanding benchmarks with a compact model.
paper_ai_keywords: self-supervised pre-training multimodal learning mask language modeling graph contrastive learning multimodal representations bounding box image features visual cues FUNSD CORD SROIE Payment benchmarks