  Distributed multi-party learning provides an effective approach for training
a joint model with scattered data under legal and practical constraints.
However, due to the quagmire of a skewed distribution of data labels across
participants and the computation bottleneck of local devices, how to build
smaller customized models for clients in various scenarios while providing
updates appliable to the central model remains a challenge. In this paper, we
propose a novel adaptive differentiable sampling framework (ADDS) for robust
and communication-efficient multi-party learning. Inspired by the idea of
dropout in neural networks, we introduce a network sampling strategy in the
multi-party setting, which distributes different subnets of the central model
to clients for updating, and the differentiable sampling rates allow each
client to extract optimal local architecture from the supernet according to its
private data distribution. The approach requires minimal modifications to the
existing multi-party learning structure, and it is capable of integrating local
updates of all subnets into the supernet, improving the robustness of the
central model. The proposed framework significantly reduces local computation
and communication costs while speeding up the central model convergence, as we
demonstrated through experiments on real-world datasets.

Machine Learning