--workers refers to my DataLoaders's number of workers, sorry for the confusion.
The following is the log file for the whole run it’s a bit of a behemoth: 32-node-fail.out · GitHub
One of the exports is incorrect so I’m running a job to fix that… But otherwise some of the print statements I include print the global rank as well as the address they are initializing to. All seems to be in order there but they get stuck during the process group initialization:
    if args.distributed:
        print(f"Starting rank {args.rank}")
        print(f"Initing to IP {os.environ['MASTER_ADDR']}")
        dist.init_process_group(
            backend=args.dist_backend,
            init_method=args.dist_url,
            world_size=args.world_size,
            rank=args.rank,
        )

If any other decorators or debugging terms would be helpful I’m glad to rerun with those attached!
Update: even with the NCCL_IB_TIMEOUT fixed back at 22 there was no success.