BERT was trained on book corpus and english wikipedia both of which are available in dataset library

  
      huggingface.co
  
  
    

Hugging Face – On a mission to solve NLP, one commit at a time. 126



  
  
    
    
  
  



  
      huggingface.co
  
  
    

Hugging Face – On a mission to solve NLP, one commit at a time. 63



  
  
    
    
  
  


Transformers has recently included dataset for for next sent prediction which you could use

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/data/datasets/language_modeling.py#L258 121

            i += 1  # go to next line
        return examples

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> Dict[str, torch.tensor]:
        return self.examples[i]


class TextDatasetForNextSentencePrediction(Dataset):
    """
    This will be superseded by a framework-agnostic approach
    soon.
    """

    def __init__(
        self,
        tokenizer: PreTrainedTokenizer,
        file_path: str,
        block_size: int,



  
  
    
    
  
  


and there’s also NSP head  for BERT

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L560 29

class BertOnlyMLMHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.predictions = BertLMPredictionHead(config)

    def forward(self, sequence_output):
        prediction_scores = self.predictions(sequence_output)
        return prediction_scores


class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score


class BertPreTrainingHeads(nn.Module):



  
  
    
    
  
  


EDIT:
BertForPreTraining class can be used for both MLM and NSP
with the current example/languae-modeling I guess it’s only possible to either use MLM or NSP, you might need to write your own script to combine these.