Thanks for @eqy’s suggestion! I find that if using the PyTorch distributed launch utility script, then there is a --master_port argument (see here 2) one can use to set the port, and once different distributed jobs are configured to use different ports, the “Address already in use” problem goes away.