Genuine question, is that “model sharding” or model parallelism implemented, yet? So far, I’ve only been doing the multi-GPU stuff with data parallelism (e.g., like https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html 60).