Solved by sgugger in post #4 
              
              
                The Trainer code will run on distributed or one GPU without any change. Regarding your other questions: 

you need to define your model in all processes, they will see different part of the data each and all copies will be kept the same.
the tokenizer and dataset preprocessing can either be done on …