Machine Learning, NLP, Bert, Albert, Transformers.
tasks. Nevertheless, multiple studies showed that it might be beneficial to get rid of the NSP objective mainly because of its simplicity, compared to MLM. Following this idea, ALBERT researchers also decided to remove the NSP task and replace it with sentence order prediction (SOP) problem whose