Hello @vblagoje. Yes , I found a couple of problems and managed to fix them.


I was not creating the text file in a correct way so the dataset class couldn’t parse it and create the examples. That was causing the error I have on the pastebin link.


The datacollator class seems to return “masked_lm_labels” which I don’t know why and what is the relevance when doing nsp. I removed that key and everything seems to be working fine.


Also I noticed that datacollator will return “next_sentence_label” instead of “labels”.That seemed a problem for the Trainer class so I changed “next_sentence_label” into “labels”.


I just saw the source code, copy it locally and made the changes locally.
I am currently getting the progress bars when training starts and I am waiting for it to finish training.