Yes, I’ve updated the code accordingly and ran with 2 GPUs (1 per node) but now I get a similar but less descriptive error:
checkpoint1 with rank 0 and world_size 0
checkpoint1 with rank 0 and world_size 0
--------------------------------------------------------------------------
Open MPI has detected that this process has attempted to initialize
MPI (via MPI_INIT or MPI_INIT_THREAD) more than once.  This is
erroneous.
--------------------------------------------------------------------------
[node024:18276] *** An error occurred in MPI_Init_thread
[node024:18276] *** reported by process [3619422209,0]
[node024:18276] *** on a NULL communicator
[node024:18276] *** Unknown error
[node024:18276] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[node024:18276] ***	and potentially your MPI job)
[fs0:19944] 1 more process has sent help message help-mpi-runtime.txt / mpi_init: invoked multiple times
[fs0:19944] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[fs0:19944] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal unknown handle

Note that the logging outputted is before the dist.init_process_group
If I perfom the basic checks in both nodes I get the expected
1.8.0a0+262bd64
1.8.0a0+262bd64
Hello from P0 on node024 out of 20
<torch.cuda.device object at 0x2aaafd2efc10>
1
GeForce GTX TITAN X
True
0
-------------------------
Hello from P1 on node025 out of 20
<torch.cuda.device object at 0x2aaafd2efd50>
1
GeForce GTX TITAN X
True
0
-------------------------