Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
makes it easy to scale nodes, and Kubernetes or cloud solutions can orchestrate them effectively. Each node can have multiple GPUs (or TPUs and other devices) and various containers in a container cluster. In the cloud-native solution, the nodes can be hidden from users. A container manages one or