title: FormNetV2: Multimodal Graph Contrastive Learning for Form Document
    Information Extraction
summary: The recent advent of self-supervised pre-training techniques has led to a
  surge in the use of multimodal learning in form document understanding.
  However, existing approaches that extend the mask language modeling to other
  modalities require careful multi-task tuning, complex reconstruction target
  designs, or additional pre-training data. In FormNetV2, we introduce a
  centralized multimodal graph contrastive learning strategy to unify
  self-supervised pre-training for all modalities in one loss. The graph
  contrastive objective maximizes the agreement of multimodal representations,
  providing a natural interplay for all modalities without special customization.
  In addition, we extract image features within the bounding box that joins a
  pair of tokens connected by a graph edge, capturing more targeted visual cues
  without loading a sophisticated and separately pre-trained image embedder.
  FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
  and Payment benchmarks with a more compact model size.
paper_authors._id: 6454624706728ff79a382477
paper_authors.name: Chen-Yu Lee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-08T08:38:20.262Z
paper_authors._id: 64d1fe784dfd5df7076bfd1a
paper_authors.avatarUrl: /avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
paper_authors.fullname: Chen-Yu Lee
paper_authors.type: user
paper_authors.user: chenyulee
paper_authors._id: 6454624706728ff79a382478
paper_authors.name: Chun-Liang Li
paper_authors._id: 6454624706728ff79a382479
paper_authors.name: Hao Zhang
paper_authors._id: 6454624706728ff79a38247a
paper_authors.name: Timothy Dozat
paper_authors._id: 6454624706728ff79a38247b
paper_authors.name: Vincent Perot
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T09:38:20.186Z
paper_authors._id: 64e3f501bbee0c24b95e20c2
paper_authors.avatarUrl: /avatars/bae7b5297e779d91da21ff50389dd90a.svg
paper_authors.fullname: Vincent
paper_authors.type: user
paper_authors.user: vperot
paper_authors._id: 6454624706728ff79a38247c
paper_authors.name: Guolong Su
paper_authors._id: 6454624706728ff79a38247d
paper_authors.name: Xiang Zhang
paper_authors._id: 6454624706728ff79a38247e
paper_authors.name: Kihyuk Sohn
paper_authors._id: 6454624706728ff79a38247f
paper_authors.name: Nikolai Glushnev
paper_authors._id: 6454624706728ff79a382480
paper_authors.name: Renshen Wang
paper_authors._id: 6454624706728ff79a382481
paper_authors.name: Joshua Ainslie
paper_authors._id: 6454624706728ff79a382482
paper_authors.name: Shangbang Long
paper_authors._id: 6454624706728ff79a382483
paper_authors.name: Siyang Qin
paper_authors._id: 6454624706728ff79a382484
paper_authors.name: Yasuhisa Fujii
paper_authors._id: 6454624706728ff79a382485
paper_authors.name: Nan Hua
paper_authors._id: 6454624706728ff79a382486
paper_authors.name: Tomas Pfister
paper_title: FormNetV2: Multimodal Graph Contrastive Learning for Form Document
    Information Extraction
paper_summary: The recent advent of self-supervised pre-training techniques has led to a
  surge in the use of multimodal learning in form document understanding.
  However, existing approaches that extend the mask language modeling to other
  modalities require careful multi-task tuning, complex reconstruction target
  designs, or additional pre-training data. In FormNetV2, we introduce a
  centralized multimodal graph contrastive learning strategy to unify
  self-supervised pre-training for all modalities in one loss. The graph
  contrastive objective maximizes the agreement of multimodal representations,
  providing a natural interplay for all modalities without special customization.
  In addition, we extract image features within the bounding box that joins a
  pair of tokens connected by a graph edge, capturing more targeted visual cues
  without loading a sophisticated and separately pre-trained image embedder.
  FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
  and Payment benchmarks with a more compact model size.
paper_ai_keywords: self-supervised pre-training multimodal learning mask language modeling centralized multimodal graph contrastive learning graph contrastive objective multimodal representations image features bounding box token graph edge image embedder FUNSD CORD SROIE Payment benchmarks compact model size