BertForPreTraining has both heads (MLM and NSP) so you’re correct with your assumption.
I recently found this script which I used to train both MLM and NSP heads.  I ultimately had one document which contained multiple paragraphs and each paragraph pertained to one topic.  I’m using each paragraph as a separate context when using extractive QA.  This ultimately gave me a list of strings where each string was a paragraph (ie. multiple sentences). This was stored inside the ‘text’ variable.
NSP Training Data & Labels
sentence_a = []
sentence_b = []
label = []

bag_of_sentences = [sentence.strip() for paragraph in text for sentence in paragraph.split('.') if sentence != '']
bag_size = len(bag_of_sentences)

for paragraph in text:
  sentences = [sentence.strip() for sentence in paragraph.split('.')]
  num_sentence = len(sentences)
  if num_sentence > 1: # if there is more than one sentence in the paragraph
    start = random.randint(0, num_sentence-2) # select a random integer b/w 0 and len(sentences-2)
    # This is "IsNext" sentence
    # selects a random float b/w 0-1 and if that float is greater than 0.5 (ie. 50%) append sentence_a w/a random sentence from the paragraph then append the next paragraph to sentence_b)
    if random.random() > 0.5: 
      sentence_a.append(sentences[start]) 
      sentence_b.append(sentences[start+1])
      label.append(0) # the label for IsNext = 0
    # IsNotNext append
    else:
      # This is IsNotNext sentence
      index = random.randint(0, bag_size-1) 
      sentence_a.append(sentences[start]) # append a random sentence from the paragraph to sentence_a but append a random sentence from the entire text to sentence_b
      sentence_b.append(bag_of_sentences[index])
      label.append(1) #the labels ofr IsNotNext = 1

from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
inputs = tokenizer(sentence_a, sentence_b, return_tensors='pt', 
                   max_length=512, truncation=True, padding='max_length')

# Creating labels for NSP
inputs['next_sentence_label'] = torch.LongTensor([label]).T

Creating 15% Random Input_Ids Masks for MLM
inputs['labels'] = inputs.input_ids.detach().clone()
rand = torch.rand(inputs.input_ids.shape)
mask_arr = (rand < 0.15) * (inputs.input_ids != 101) * (inputs.input_ids != 102) * (inputs.input_ids != 0)

selection = []
for i in range(inputs.input_ids.shape[0]):
  selection.append(torch.flatten(mask_arr[i].nonzero()).tolist())

for i in range(inputs.input_ids.shape[0]):
  inputs.input_ids[i, selection[i]] = 103

This will give you the training data required for further pre-train a bert model.  I just used the ‘bert-base-uncased’ model.
My question for you is how do I further pre-train this model from a chosen checkpoint?