Hi @mrshenli,
Thanks for that example, I was looking all over online for something like that but couldn’t find one.
If anyone else is curious, I used a similar slurm variable to get the worker ID’s and wrote them to a file in my shell script:
touch /dev/null > $WORKDIR/mynodes.txt
scontrol show hostname $SLURM_JOB_NODELIST > $WORKDIR/mynodes.txt

Then I simply read from this file in my pytorch script.
To get the rank 0 IP address, I did os.environ['MASTER_ADDR'] = os.system(nodes_list[0] + ' -I awk {print$1}') after reading the rank 0 node ID from my written file. For my specific cluster, the port number didn’t matter, I just picked 8080.
For data parallelism, I’m planning on simply using the distributed library which requires the same MASTER_ADDR, so not expecting a problem there.