As you can see from the error message, it’s better to use multiple process for multiple GPU training even on a single node, you can use torch.multiprocessing.spawn(train_fn, args=(world_size,), nprocs=world_size) to initialize the training in multiple process