churchillmic:

No, the SLURM system gives you a unique SLURM_JOB_ID for each run that you do (which is why I’m using it, to ensure the file is unique for each run).


Thanks, that rules out clobbering the same file from multiple runs.



 churchillmic:

I noticed the note on fcntl, is there some test I should run on the parallel file system to ensure there are no issues with correct locking?


There is. We have a test for the file store that’s built by default if you compile from source and will be located at build/bin/FileStoreTest. This test automatically creates some files in TMPDIR, which you can override yourself to force it to use the GPFS path. This doesn’t fully simulate the scenario you have with multiple machines, but at least hammers the file system with multiple processes from the same machine. It could uncover something, so it’s definitely worth a try.



 churchillmic:

I was able to try out NCCL, and this appears to be working for the > 2 nodes runs, so its not as urgent, but I’d still be interested in figuring this out


The use of this store when using the NCCL backend is very light. Only a single process writes to the file and all others read from it. When using the Gloo backend, everybody both writes to and reads from the file, causing a lot more contention.