i’m definitely missing something…curious to hear your views on what that is.
based on the HF documentation i thought it would be possible to simply pass an in memory dataframe:
unlabelled_dataset = Dataset.from_pandas(unlabelled_corpus)    #create Dataset object for tokenization
take the relevant paragraphs column from the dataframe that i want to finetune the pretrained model on.  this column has a few sentences or a paragraph on each row of the dataframe.  and then pass it to the pretrained tokenizer after intialization:
tokenizerBERT = BertTokenizerFast.from_pretrained(‘bert-base-uncased’)  #BERT model tokenization & check
encodingBERT = tokenizerBERT(unlabelled_dataset[‘paragraphs’], padding=True, truncation=True)
initializing the data collator w/ reference to the pretrained tokenizer:
data_collator_BERT = DataCollatorForLanguageModeling(tokenizer=tokenizerBERT, mlm=True, mlm_probability=0.15)
and then initializing the model, training arguements and trainer before calling it would do the trick:
model_BERT = BertForMaskedLM.from_pretrained(‘bert-base-uncased’)
training_args_BERT = TrainingArguments(
output_dir=’./BERT’,
num_train_epochs=10,
evaluation_strategy=‘steps’,
warmup_steps=10000,
weight_decay=0.01,
per_device_train_batch_size=64,
)
trainer_BERT = Trainer(
model=model_BERT,
args=training_args_BERT,
data_collator=data_collator_BERT,
train_dataset=encodingBERT,
)
trainer_BERT.train()
trainer.save_model(’./models/royalBERT’)
the features of the loaded dataset.from_pandas are provided below.  i only am looking to train the model on the paragraphs column hence why i pass that column in the called pretrained tokenizer.
{‘index’: Value(dtype=‘string’, id=None), ‘filename’: Value(dtype=‘string’, id=None), ‘page-id’: Value(dtype=‘string’, id=None), ‘paragraph-id’: Value(dtype=‘string’, id=None), ‘paragraphs’: Value(dtype=‘string’, id=None), ‘index_level_0’: Value(dtype=‘int64’, id=None)}