Thanks for your kind reply.
I put the code and error message in this open issue on github.https://github.com/huggingface/transformers/issues/6616 36  My problem is mainly about ‘index out of range in self’ error, given that I alredy set a very low max_length value in tokenizer. I was also confused using native PyTorch to train before, in which I don’t know how to handle mask and pass labels to model. But if DataCollatorForLanguageModeling and Trainer works well, that doesn’t matter.
Could you help verify if there is any mistake in my piece of code? Thank you!