Take a look at this NeurIPS 2021 experiment:

  
      

      Training Transformers Together
  

  
    

Train vast neural networks together 1

  A NeurIPS'21 demonstration that explains how to train large models together with multiple collaborators.