I can’t comment on all libs, but #1 vs. #5:
apex.DDP is deprecated in favor of the native PyTorch DistributedDataParallel. Similar to the mixed-precision training (amp) apex provided DDP while there shouldn’t be any advantage of using it anymore and you should thus switch the the native DDP implementation.