Hi, I have been trying to train BERT from scratch using the wonderful hugging face library. I am referring to the Language modeling tutorial and have made changes to it for the BERT.  As I am running on a completely new domain I have trained my own tokenizer, which trains fine. However, I run into the following error during the training of the model.
)
/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in train(self, model_path, trial)
710         self.state.is_world_process_zero = self.is_world_process_zero()
711
–> 712         tr_loss = torch.tensor(0.0).to(self.args.device)
713         self._logging_loss_scalar = 0
714         self._total_flos = self.state.total_flos
RuntimeError: CUDA error: device-side assert triggered
From what I understand this is related to some tensor mismatch, however, I unable to resolve this and can’t understand where I am going wrong during the model building. I would really appreciate your help with this.
I have attached the colab notebook for reference
(https://colab.research.google.com/drive/12NHfXeUBo7RBl3Kffa-715i-Zpd0MOzP?usp=sharing 183).
I am using BertForPreTraining, TextDatasetForNextSentencePrediction, DataCollatorForNextSentencePrediction and BertFasttokeizer.
Environment:
tokenizers: 0.9.2
transformers: 3.4.0
torch: 1.7.0
CUDA: 10.1