It’s pretty similar - we all_gather the metadata, indices, and values, and then each node does a local sum of the sparse tensors. Here’s the implementation:  https://github.com/pytorch/pytorch/blob/65bd38127a34d428915c88507878b4735edf005f/torch/lib/c10d/ProcessGroupGloo.cpp#L939 1