Hi all,
I’m suffering from the hugh bandwidth requirement when I train my model in multiple nodes, so considering to apply this 3. However, I have no idea that what part of the pytorch code I have to replace. I guess that I need to make another torch.distributed.reduce_op to replace torch.distributed.reduce_op.SUM, but I couldn’t find any guideline to implement another operator from official docs or tutorials. Could you let me know what part of pytorch code I have to refer to?
Thanks,
Jinserk