y91:

horovod


Since you are using horovod and mpi, this can be a bug in either horovod, pytorch distributed code or mpi. The first two seem more likely. If you can come up with a more reliable short script, I suggest you post to GitHub repos of those two projects.