Distributed Weight Consolidation: A Brain Segmentation Case Study
Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individual sites can be shared and combined with fewer restrictions. Training on distributed data and combining the resulting networks is often viewed as continual learning, but these methods require networks to be trained sequentially. In this paper, we introduce distributed weight consolidation (DWC), a continual learning method to consolidate the weights of separate neural networks, each trained on an independent dataset. We evaluated DWC with a brain segmentation case study, where we consolidated dilated convolutional neural networks trained on independent structural magnetic resonance imaging (sMRI) datasets from different sites. We found that DWC led to increased performance on test sets from the different sites, while maintaining generalization performance for a very large and completely independent multi-site dataset, compared to an ensemble baseline.
Research is becoming an increasingly collaborative endeavor but experiments that result in data collection are still costly. The costs, together with privacy, regulations and practicality concerns lead to the current situation when large datasets are amassed across multiple research groups yet any single group rarely has enough data to train a machine learning algorithm. The paper proposes an algorithm for training machine learning models in these settings in the continual learning style, when a model is first trained on data at some of the sites and then the learned weights are aggregated and subsequent models are fine-tuned from that point. This is achieved thanks to a variational approximation and independence assumption on filter weights  My two major concerns are  1.  The unrealistic testing setup that goes against the offered decentralized research centers models. The test in the paper are conducted on a very few sites with a very large amount of data, and not on many sites with moderate data sizes. To gain an intuition whether the approach works one would need many more realistic setups for validation.  1.  The choice of brain segmentation problem to demonstrate PWC. It is a well known feature of most of the neural segmentation methods that unlike other deep learning algorithms they only need a few labeled examples, since each pixel/voxel is labeled and provides a signal. Thus, among all of the deep learning algorithms the paper seems to imply it will improve the chosen method seems to be the one benefiting the least. Furthermore, it would be good to see and compare with performance of the original MeshNet segmentation algorithm just on the data that was used at the final stages. I suspect, it may perform as well.  Other concerns:  1.  line 52, in fact, MeshNet, when ran uses quite a lot of memory, may be even more than other mentioned methods, but that memory is in the activations, not in useful weights. So may be the paper needs to be more explicit of the fact, that MeshNet is the only one of the mentioned models that is feasible to use in the proposed settings as all other models require transfer of Gigabytes of parameters for each trained model, while MeshNet's parameters amount to under a Megabyte? 2.  The cited papers of segmentation techniques for brain imaging are just the most recent. PyramidLSTM from 2015 is not cited, for example. 3.  It is unclear from the text whether the algorithm ships the weights over the network or what happens to them at all. it would be good to have that explained. 4.  Line 77-78 - editorial comment is left in the manuscript. 5.  independent values of w in spatial filter are hard to assign the meaning to. The whole purpose of the local spatial filter is to capture correlations in the input, so the weights cannot be independent. 6.  The notation is heavy and is not optimized for getting the point across. The use of indexing is excessive and leads to difficulties with interpreting even simple mathematics. Equation 16 is an example that could be greatly simplified but then it would took quite simple. 7.  Line 141 implies that parameter uncertainty increases with each consolidation, If that is try then does not that defeat the point of the work? 8.  The model uses a truncated set of FreeSurfer units, while the complete 104 ROI set in the atlas could be more useful for practicioners. 9.  line 153 provides citations for splitting each volume, but the provided references did not split, but sample subvolumes. 10. Lines 158-159 needs to qantified. 11. lines 171-173 the vanilla MeshNet should be also among the baselines. 12. it may be worth citing prior work on successfully executing the model that the authors describe here (ENIGMA 1) and potentially any of the papers that worked out the decentralized data setup in detail, including a similar to the proposed "hot potato" algorithm schema as well as the iterative one (COINSTAC papers 2). Without those citations the current language in the paper misleads the reader to think that the specific decentralized biomedical setup is also proposed by the authors of this paper.     1.  Thompson, P. M. and others (2014). The ENIGMA consortium     2.  Plis, SM et al., 2016. COINSTAC: a privacy enabled model  1.  lines 43-45 imply that the datasets were combined but line 153 states otherwise 2.  Lines 199-200 The vision does come across clear. 3.  line 203 - unsupported claim. 4.  Why the subvolumes size have been decreased from 38 to 32? 5.  The receptive field size of convolutional network is 37, which is bigger then 32 subvolume size  Minor:  -   line 94 - such elastic, should be such as elastic -   line 95 - reinterpretted - an extra t