Hey there, I’ll share all the details in a week or so. Until I really wrap this up note that I used this script 30 to create sharded datasets for bert training. After dataset preparation, I used this script 58 to train BERT. There are still a few small bugs to iron out but it works quite well. I can train bert base in about 8-9 hours on 8gpu machine using Pytorch distributed training.
HTH,
Vladimir