rvarm1:

Although I’m a little confused, while the barrier at any of your points should work, I’m not sure how it helps you use only one GPU for validation? A barrier will just block all processes until all processes have entered the barrier.


I am also confused about this. My thought process is just that it seems like a waste of power to do the same validation step on all GPUs at the same time.
Right now, my validation is coded to be done by a single process (and by consequence a single GPU), so there wouldn’t be any performance gain running it across multiple GPUs.
I will try to reproduce the error and post an issue on Monday, since currently I don’t have access to the machine.
I guess in the end I will have to just run validation on all GPUs.