In federated learning problems, data is scattered across different servers and exchanging or pooling it is often impractical or prohibited. We develop a Bayesian nonparametric framework for federated learning with neural networks. Each data server is assumed to train local neural network weights, which are modeled through our framework. We then develop an inference approach that allows us to synthesize a more expressive global network without additional supervision or data pooling. We then demonstrate the efficacy of our approach on federated learning problems simulated from two popular image classification datasets.
Summary: The paper considers federate learning of neural networks, i.e. data are distributed on multiple machines and the allocation of data points is potentially inhomogenous and unbalanced. The paper proposes a method to combine multiple networks trained locally on different data shards and form a global neural network. The key idea is to identify and reuse neurons that can be used for all shards and add new neurons to the global network if necessary. The matching and combination process is done via MAP inference of a model using a Beta-Bernoulli process. Some experiments on federated learning demonstrate the performance of the proposed method.
General evaluation (+ pro/ - con, more specific comments/questions below):
+ the paper is very well-written -- the BBP presentation is light but very accessible. The experimental set up seems sound.
+ the matching procedure is novel for federated training of neural networks, as far as I know, but might not be if you are a Bayesian nonparametric person, as the paper pointed out similar techniques have been used for topic models.
- the results seem to back up the claim that the proposed is a good candidate for combining networks at the end of training, but the performance is very similar or inferior to naive combination methods and that the global network is way larger than individual local network and nearly as large as simply aggregating all neurons together.
- the comparison to recent federated learning methods is lacking (e.g. McMahan et al, 2017) (perhaps less communication efficient than the proposed method, but more accurate).
Specific comments/questions/suggestions:
- the MAP update for the weights given the assignment matrix is interesting and resembles exactly how the Bayesian committee machine algorithm of Tresp (2000) works, except that the variances are not learnt for each parameter but fixed for each neuron. On this, there are several hyperparameters for the model, e.g. variance sigma_j -- how are these tuned/selected?
- the local neural networks are very mall (only 50 neurons per layer). How do they perform on the test set on the homogeneous case? Is there a performance loss by combining these networks together?
- the compression rate is not that fantastic, i.e. the global network tends to add new neuron for each local neuron considered. Is this because it is in general very hard to identify similar neuron and group them together? In the homogeneous case, surely there are some neurons that might be similar. Or is it because of the MAP inference procedure/local optima?