Machine Learning, Mpi, Pytorch, TensorFlow.
optimization was introduced for this use-case — run client for each replica worker and have each client construct a slightly different computation graph, only containing their replica ops. This makes this use-case practical but makes the framework harder to understand. Generally when optimizing