Noise-Aware Training of Layout-Aware Language Models
A visually rich document (VRD) utilizes visual features along with linguistic
cues to disseminate information. Training a custom extractor that identifies
named entities from a document requires a large number of instances of the
target document type annotated at textual and visual modalities. This is an
expensive bottleneck in enterprise scenarios, where we want to train custom
extractors for thousands of different document types in a scalable way.
Pre-training an extractor model on unlabeled instances of the target document
type, followed by a fine-tuning step on human-labeled instances does not work
in these scenarios, as it surpasses the maximum allowable training time
allocated for the extractor. We address this scenario by proposing a
Noise-Aware Training method or NAT in this paper. Instead of acquiring
expensive human-labeled documents, NAT utilizes weakly labeled documents to
train an extractor in a scalable way. To avoid degradation in the model's
quality due to noisy, weakly labeled samples, NAT estimates the confidence of
each training sample and incorporates it as uncertainty measure during
training. We train multiple state-of-the-art extractor models using NAT.
Experiments on a number of publicly available and in-house datasets show that
NAT-trained models are not only robust in performance -- it outperforms a
transfer-learning baseline by up to 6% in terms of macro-F1 score, but it is
also more label-efficient -- it reduces the amount of human-effort required to
obtain comparable performance by up to 73%.
660c2763a3ffe3b8f9384ef7
Ritesh Sarkhel
660c2763a3ffe3b8f9384ef8
Xiaoqi Ren
660c2763a3ffe3b8f9384ef9
Lauro Beltrao Costa
660c2763a3ffe3b8f9384efa
Guolong Su
660c2763a3ffe3b8f9384efb
Vincent Perot
660c2763a3ffe3b8f9384efc
Yanan Xie
660c2763a3ffe3b8f9384efd
Emmanouil Koukoumidis
660c2763a3ffe3b8f9384efe
Arnab Nandi
Noise-Aware Training of Layout-Aware Language Models
A visually rich document (VRD) utilizes visual features along with linguistic
cues to disseminate information. Training a custom extractor that identifies
named entities from a document requires a large number of instances of the
target document type annotated at textual and visual modalities. This is an
expensive bottleneck in enterprise scenarios, where we want to train custom
extractors for thousands of different document types in a scalable way.
Pre-training an extractor model on unlabeled instances of the target document
type, followed by a fine-tuning step on human-labeled instances does not work
in these scenarios, as it surpasses the maximum allowable training time
allocated for the extractor. We address this scenario by proposing a
Noise-Aware Training method or NAT in this paper. Instead of acquiring
expensive human-labeled documents, NAT utilizes weakly labeled documents to
train an extractor in a scalable way. To avoid degradation in the model's
quality due to noisy, weakly labeled samples, NAT estimates the confidence of
each training sample and incorporates it as uncertainty measure during
training. We train multiple state-of-the-art extractor models using NAT.
Experiments on a number of publicly available and in-house datasets show that
NAT-trained models are not only robust in performance -- it outperforms a
transfer-learning baseline by up to 6% in terms of macro-F1 score, but it is
also more label-efficient -- it reduces the amount of human-effort required to
obtain comparable performance by up to 73%.
Noise-Aware Training (NAT) method improves efficiency and robustness in training named entity extractors using weakly labeled documents, outperforming transfer learning in terms of macro-F1 score and reducing human labeling effort.
Noise-Aware Training NAT weakly labeled documents named entity extractors macro-F1 score human labeling effort