I noticed the note on fcntl, is there some test I should run on the parallel file system to ensure there are no issues with correct locking? I think GPFS should be fine, but perhaps theres some edge case, and a newer driver or something caused things to mess up.
I was able to try out NCCL, and this appears to be working for the > 2 nodes runs, so its not as urgent, but I’d still be interested in figuring this out