Machine Learning, Mpi, Pytorch, TensorFlow.
continue when one worker pauses or dies is next to impossible. Ray approach provides the most control at the cost of verbosity — you could have have different actors exchanging messages and implement any kind of aggregation logic in pure Python. In particular, the various aggregation methods like