Thank you so much for the quick reply - sorry I think I phrased it poorly. In run_mlm.py - there are 500+ lines of code. We have our own python scripts that load some data, pass it to a Huggingface trainer, etc. very similar to the Seq2SeqTrainer notebook I have linked above. These are like 10 lines of code… for one GPU. My question is, to convert a very basic Huggingface python script (loads a model, loads a dataset, passes the dataset and model to the trainer)… what is the bare minimum to “convert” this code, which runs on one GPU using huggingface trainer, to multi gpu using huggingface trainer? Said differently… if you take your run_mlm.py that you linked to… is there a much simpler version of that somewhere that doesn’t have all the argument checking, etc… just the bare minimum to train something on multi-GPU using torch.distributed and a huggingface trainer?
Going more into the specifics - here is the bulk of the code I am trying to convert to multi GPU:
trainer = transformers.Seq2SeqTrainer(
    model=model,
    tokenizer=fa,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=transformers.default_data_collator,
)

model.train()
trainer.train()

My specific questions - understanding that when I run this using the distributed launcher, it is running 8 of these (for 8 GPUs):


My “model” object is created above that code using from_encoder_decoder_pretrained - if it is running 8 times… is that making 8 copies of the model or is the huggingface function from_encoder_decoder_pretrained handling that for me?


my tokenizer is created similarly using the from_pretrained function from Huggingface - is it making 8 copies or does it deal with it?


train_dataset & eval_dataset is a very simple dataset class I wrote myself - it just gets a row out of a dataframe that contains a file name, and then loads the file, and returns the file. I am not worried about 8 of these running in parallel - they are thread safe because they are just reading files - but… same question… are there 8 of these running at the same time? Should I care and try and make only one of them?


Hopefully that makes more sense and explains why I am looking for some basic boilerplate. Please by all means push back and correct me!!! Have learned a lot!!