Hi vblagoje,
I am new to transformer. I have been playing the hugging face model for several month and I think I am thinking to made a some small changes on the Bert Model and pretrain it from scratch. I saw you discussing on another post several days ago about the pretraining process. I was wondering if you know the pretraining repository made by Nvidia?

  
      
      GitHub
  
  
    

NVIDIA/DeepLearningExamples 20

Deep Learning Examples. Contribute to NVIDIA/DeepLearningExamples development by creating an account on GitHub.


  
  
    
    
  
  


I think they implemented the lamb optimizer, NSP objective and wrote code to better utilized multiple gpu during distributed training. I still haven’t use it yet because I have some trouble with installing docker  on the remote machine I am working on. I was just wondering if you already seen this repository or tried it, or if you have any advice on pretraining bert from scratch?