If you are using DDP (DistributedDataParallel()) then you can simply save them like you do without DDP (using torch.save), because every process (i.e. GPU) will run it. Use gpu index to prevent multiple saving
of the same name