You are setting world_size=2 and have nproc_per_node=2 so you actually have 4 processes total. For each of the nodes, it looks like you are setting the device to local_rank then spawning 2 subprocesses to load the data. I don’t see anything off about what you are doing but just wanted to make sure that was what was intended.
I am not able to run the example you provided due to missing arg parsing, imports, etc. Do you have a sample script I can run locally to reproduce the error?