I’m assuming you’re referring to local_rank mentioned here: https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py 424

You should use rank and not local_rank when using torch.distributed primitives (send/recv etc). local_rank is passed to the training script only to indicate which GPU device the training script is supposed to use.
You should always use rank.

local_rank is supplied to the developer to indicate that a particular instance of the training script should use the “local_rank” GPU device. For illustration, in the example above provided by @spanev, p1 is passed local_rank 0 indicating it should use GPU device id 0.