Hi All,
I am a researcher in LBL interested in implementing distributed model parallelism in PyTorch. This could in fact be useful for our research as well. Currently, I am looking at the DistributedDataParallel classes to see how PyTorch decomposes data internally across machines.
I wonder if the PyTorch community would be interested in this and if there’s already some work on this topic.
Thank you,
Saliya