  Pre-training a language model and then fine-tuning it for downstream tasks
has demonstrated state-of-the-art results for various NLP tasks. Pre-training
is usually independent of the downstream task, and previous works have shown
that this pre-training alone might not be sufficient to capture the
task-specific nuances. We propose a way to tailor a pre-trained BERT model for
the downstream task via task-specific masking before the standard supervised
fine-tuning. For this, a word list is first collected specific to the task. For
example, if the task is sentiment classification, we collect a small sample of
words representing both positive and negative sentiments. Next, a word's
importance for the task, called the word's task score, is measured using the
word list. Each word is then assigned a probability of masking based on its
task score. We experiment with different masking functions that assign the
probability of masking based on the word's task score. The BERT model is
further trained on MLM objective, where masking is done using the above
strategy. Following this standard supervised fine-tuning is done for different
downstream tasks. Results on these tasks show that the selective masking
strategy outperforms random masking, indicating its effectiveness.

Computation and Language