CCL:

So far I’ve worked out that the line dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url, world_size=1, rank=args.rank) intialises the same process on all 8 GPUs


The init_process_group API only sets up the process where this function is invoked. And, as the world_size is set to 1, It only expects one process in the distributed training gang. If you would like to use multiple processes, please see this example 20.

but this causes problems later in my code, where I need to get the specific GPU machine index.

To get machine index, will it work if you use args.rank?

I tried to do this with  torch.cuda.current_device()  but it also returns 0 despite  nvidia-smi  showing that all 8 GPUs have been used.

The torch.cuda.current_device()  returns the current device. By default, it is the first GPU, which is indeed indexed by 0.