Thanks for the clarifications, reading through the github issues it seems that:


local_rank  is actually the ID  within  a worker; multiple workers have a  local_rank  of  0 , so they’re probably trampling each other’s checkpoints.


added a  --global_rank  command line argument as well. (solution)


someone else comments, torch.distributed.launch  sets up a  RANK  environment variable 4 which can be used to detect if you are on the master process (with  os.environ['RANK'] == '0'  from python


you can use  torch.distributed.get_rank()  to get the global rank. (I suppose this might be the most appropriate way to do it?)