title: Noise-Aware Training of Layout-Aware Language Models
summary: A visually rich document (VRD) utilizes visual features along with linguistic
  cues to disseminate information. Training a custom extractor that identifies
  named entities from a document requires a large number of instances of the
  target document type annotated at textual and visual modalities. This is an
  expensive bottleneck in enterprise scenarios, where we want to train custom
  extractors for thousands of different document types in a scalable way.
  Pre-training an extractor model on unlabeled instances of the target document
  type, followed by a fine-tuning step on human-labeled instances does not work
  in these scenarios, as it surpasses the maximum allowable training time
  allocated for the extractor. We address this scenario by proposing a
  Noise-Aware Training method or NAT in this paper. Instead of acquiring
  expensive human-labeled documents, NAT utilizes weakly labeled documents to
  train an extractor in a scalable way. To avoid degradation in the model's
  quality due to noisy, weakly labeled samples, NAT estimates the confidence of
  each training sample and incorporates it as uncertainty measure during
  training. We train multiple state-of-the-art extractor models using NAT.
  Experiments on a number of publicly available and in-house datasets show that
  NAT-trained models are not only robust in performance -- it outperforms a
  transfer-learning baseline by up to 6% in terms of macro-F1 score, but it is
  also more label-efficient -- it reduces the amount of human-effort required to
  obtain comparable performance by up to 73%.
paper_authors._id: 660c2763a3ffe3b8f9384ef7
paper_authors.name: Ritesh Sarkhel
paper_authors._id: 660c2763a3ffe3b8f9384ef8
paper_authors.name: Xiaoqi Ren
paper_authors._id: 660c2763a3ffe3b8f9384ef9
paper_authors.name: Lauro Beltrao Costa
paper_authors._id: 660c2763a3ffe3b8f9384efa
paper_authors.name: Guolong Su
paper_authors._id: 660c2763a3ffe3b8f9384efb
paper_authors.name: Vincent Perot
paper_authors._id: 660c2763a3ffe3b8f9384efc
paper_authors.name: Yanan Xie
paper_authors._id: 660c2763a3ffe3b8f9384efd
paper_authors.name: Emmanouil Koukoumidis
paper_authors._id: 660c2763a3ffe3b8f9384efe
paper_authors.name: Arnab Nandi
paper_title: Noise-Aware Training of Layout-Aware Language Models
paper_summary: A visually rich document (VRD) utilizes visual features along with linguistic
  cues to disseminate information. Training a custom extractor that identifies
  named entities from a document requires a large number of instances of the
  target document type annotated at textual and visual modalities. This is an
  expensive bottleneck in enterprise scenarios, where we want to train custom
  extractors for thousands of different document types in a scalable way.
  Pre-training an extractor model on unlabeled instances of the target document
  type, followed by a fine-tuning step on human-labeled instances does not work
  in these scenarios, as it surpasses the maximum allowable training time
  allocated for the extractor. We address this scenario by proposing a
  Noise-Aware Training method or NAT in this paper. Instead of acquiring
  expensive human-labeled documents, NAT utilizes weakly labeled documents to
  train an extractor in a scalable way. To avoid degradation in the model's
  quality due to noisy, weakly labeled samples, NAT estimates the confidence of
  each training sample and incorporates it as uncertainty measure during
  training. We train multiple state-of-the-art extractor models using NAT.
  Experiments on a number of publicly available and in-house datasets show that
  NAT-trained models are not only robust in performance -- it outperforms a
  transfer-learning baseline by up to 6% in terms of macro-F1 score, but it is
  also more label-efficient -- it reduces the amount of human-effort required to
  obtain comparable performance by up to 73%.
paper_ai_keywords: Noise-Aware Training (NAT) Named entities Textual modalities Visual modalities Pre-training Fine-tuning Weakly labeled documents Confidence estimation Uncertainty measure State-of-the-art Macro-F1 score