In this case, the inter-node all_to_all_single() call is NOT concurrent in the SAME device. It’s concurrent in multiple devices. There are 8 processes groups doing all_to_all_single() in parallel using following ranks:
create_inter_node_all_to_all_process_group(): local_rank = 0, global_rank = 8, ranks = [0, 8]
create_inter_node_all_to_all_process_group(): local_rank = 1, global_rank = 8, ranks = [1, 9]
…
create_inter_node_all_to_all_process_group(): local_rank = 7, global_rank = 8, ranks = [7, 15]