Thank you for both answers, Rohan! Before responding
I was trying to verify that moving the model save to after
the optimizer step solved the issue. And I didn’t have a
chance for that yet.
The oddity with the out-of-synch processes is that they run in lockstep
for quite a while, meaning several minutes. And then one of
them slows way down. Often, once the faster process finishes,
the slow one sits with 100% of both GPU and CPU (according to top).
So something is seriously wrong.
What confuses me is that I thought DDP will force synchronization
as part of the backward() call to ensure the same gradients
being used on all the processes going forward. So how could two
processes get out of sync at all even just beyond a single train loop.
Nobody is running on the machine other than these two processes.
Inputs should be of the same lengths; the model.join() attempt was
just out of desperation. I run with drop-last in the dataloader, so even
the final batch is fully populated. If anything, the two processes are too
decoupled, rather than one waiting for the other due to unequal inputs.
The one difference from all the tutorial setups is that I am using
k-fold cross validation with a distributed sampler. So epochs are
put together by train/validate over multiple splits (i.e. rotating the
validation folds) But to the DDP mechanism that shouldn’t make a difference,
I would think.
For now both processes run on the same machine to exclude NCCL version mismatch.
Though my goal is cross-machine work. I did try updating NCCL and cuda to (I think)
version 11? But this measure did not fix the issue.
Does any of the above point to a misunderstanding on my part? Of course,
it could simply be a bug in the code somewhere.
Andreas