  With the success of large-scale pretraining in NLP, there is an increasing
trend of applying it to the domain of life sciences. In particular, pretraining
methods based on DNA sequences have garnered growing attention due to their
potential to capture generic information about genes. However, existing
pretraining methods for DNA sequences largely rely on direct adoptions of BERT
pretraining from NLP, lacking a comprehensive understanding and a specifically
tailored approach. To address this research gap, we first conducted a series of
exploratory experiments and gained several insightful observations: 1) In the
fine-tuning phase of downstream tasks, when using K-mer overlapping
tokenization instead of K-mer non-overlapping tokenization, both overlapping
and non-overlapping pretraining weights show consistent performance
improvement.2) During the pre-training process, using K-mer overlapping
tokenization quickly produces clear K-mer embeddings and reduces the loss to a
very low level, while using K-mer non-overlapping tokenization results in less
distinct embeddings and continuously decreases the loss. 3) Using overlapping
tokenization causes the self-attention in the intermediate layers of
pre-trained models to tend to overly focus on certain tokens, reflecting that
these layers are not adequately optimized. In summary, overlapping tokenization
can benefit the fine-tuning of downstream tasks but leads to inadequate
pretraining with fast convergence. To unleash the pretraining potential, we
introduce a novel approach called RandomMask, which gradually increases the
task difficulty of BERT-like pretraining by continuously expanding its mask
boundary, forcing the model to learn more knowledge. RandomMask is simple but
effective, achieving top-tier performance across 26 datasets of 28 datasets
spanning 7 downstream tasks.

Artificial Intelligence;Computation and Language;Machine Learning