Many information retrieval and natural language processing  problems can be formalized as a semantic matchingtask. However, the existing work in this area has been focused in large part on the matching between short texts like finding answer spans, sentences and passages given a query or a natural language question. Semantic matching between long-form texts like documents, which can be applied to applications such as document clustering, news recommendationand related article recommendation  , is relatively less explored and needs more research effort. In recent years, self-attention based models like Transformers and BERT have achieved state-of-the-art performance in several natural language understanding tasks. These kinds of models, however, are still restricted to short text sequences like sentences due to the quadratic computational time and space complexity of self-attention with respect to the input sequence length. In this paper, we address these issues by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for document representation learning and matching, which contains several novel design choices to adapt self-attention models for long text inputs. For model pre-training, we propose the masked sentence block language modeling task in addition to the original masked word language modeling task used in BERT, to capture sentence block relations within a document. The experimental results on several benchmark data sets for long-form document matching show that our proposed SMITH model outperforms the previous state-of-the-art Siamese matching models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT for long-form document matching, and increases the maximum input text length from 512 to 2048 when compared with BERT-based baseline methods .
Many natural language processing and information retrieval problems can be formalized as the task of semantic matching. Existing work in this area has been largely focused on matching between short texts (e.g., question answering), or between a short and a long text (e.g., ad-hoc retrieval). Semantic matching between long-form documents, which has many important applications like news recommendation, related article recommendation and document clustering , is relatively less explored and needs more research effort. In recent years, self-attention based models like Transformers and BERT have achieved state-of-the-art performance in the task of text matching. These models, however, are still limited to short text like a few sentences or one paragraph due to the quadratic computational complexity of self-attention with respect to input text length. In this paper, we address the issue by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for long-form document matching. Our model contains several innovations to adapt self-attention models for longer text input. In order to better capture sentence level semantic relations within a document, we pre-train the model with a novel masked sentence block language modeling task in addition to the masked word language modeling task used by BERT. Our experimental results on several benchmark datasets for long-form document matching show that our proposed SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT . Comparing to BERT based baselines, our model is able to increase maximum input text length from 512 to 2048. We will open source a Wikipedia based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching .
D
information retrieval and
A
and information retrieval
R
a semantic matchingtask. However, the existing
the task of semantic matching. Existing
R
focused in large part on the
largely focused on
R
like finding answer spans, sentences and passages given a query or a natural language question.
(e.g., question answering), or between a short and a long text (e.g., ad-hoc retrieval).
D
texts like
R
can be applied to applications such as document clustering, news recommendationand
has many important applications like news recommendation,
A
and document clustering
R
several natural language understanding tasks. These kinds of
the task of text matching. These
R
restricted
limited
R
sequences like sentences
like a few sentences or one paragraph
D
time and space
R
the input sequence
input text
R
these issues
the issue
R
document representation learning and matching, which contains several novel design choices
long-form document matching. Our model contains several innovations
R
long text inputs. For model pre-training, we propose the
longer text input. In order to better capture sentence level semantic relations within a document, we pre-train the model with a novel
D
original
R
in BERT, to capture sentence block relations within a document. The
by BERT. Our
R
data sets
datasets
D
Siamese matching
R
for long-form document matching, and increases the
. Comparing to BERT based baselines, our model is able to increase
R
2048 when compared with BERT-based baseline methods
2048. We will open source a Wikipedia based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching