Thanks for the fast reply @rvarm1, I appreciate it.

I think the SIGSEGV is coming from dist.destroy_process_group() since the  print(f'end of training loop rank: {rank_gpu}') is logged, but not the print('finished ddp training!').
I removed dist.init_process_group(), DistributedSampler(), SyncBatchNorm, and DistributedDataPrallel() in train_model_distributed(), but kept the mp.spawn() (if that is what you meant). My jobs have been queued for a while now. I will get back at you as soon as they complete.

UPDATE: SIGSEGV occurs if I remove all the distributed training parts (see 2.).