It shouldn’t be hard to convert BertForNextSentencePrediction to use datasets. I played with wikipedia dataset for english just now. Each dataset entry is an article/document and it needs to be sentence tokenized in BertForNextSentencePrediction . Book corpus dataset entries seem to be sentences already. Let me know about your progress.