Median Selection Subset Aggregation for Parallel Inference
For massive data sets, efficient computation commonly relies on distributed algorithms that store and process subsets of the data on different machines, minimizing communication costs. Our focus is on regression and classification problems involving many features. A variety of distributed algorithms have been proposed in this context, but challenges arise in defining an algorithm with low communication, theoretical guarantees and excellent practical performance in general settings. We propose a MEdian Selection Subset AGgregation Estimator (message) algorithm, which attempts to solve these problems. The algorithm applies feature selection in parallel for each subset using Lasso or another method, calculates the 'median' feature inclusion index, estimates coefficients for the selected features in parallel for each subset, and then averages these estimates. The algorithm is simple, involves very minimal communication, scales efficiently in both sample and feature size, and has theoretical guarantees. In particular, we show model selection consistency and coefficient estimation efficiency. Extensive experiments show excellent performance in variable selection, estimation, prediction, and computation time relative to usual competitors.
This paper presents a new method to scale up training of large-scale linear models, where the tacit assumption is that the parameter set fits in a single machine but not all training instances do. The idea starts by splitting the training set into m disjoint subsets and allocating each to a different machine. Then sparsity-inducing regularised risk minimisation (such as lasso or GIC) is run independently in each machine, thus recovering a support set of features in each machine (say s_im=1 if feature i in machine m survived the selection and 0 otherwise). Then a critical step: find a consensus support by picking for each feature the median of its survival indicator value over all machines (ie compute median_m s_im for all features i). Then run OLS with those features independently in each machine and average the resulting models.This simple, elegant and intuitive approach seems to yield impressive results. The accuracy of both support recovery and prediction are on par with bootstrap lasso, while being much more scalable (constant rather than linear since embarrassingly paralellizable). I'm also impressed by the quality and clarity of presentation. The theoretical results presented are precisely those that matter: accuracy of support recovery and accuracy of the estimator.Empirical results are presented both for suitably designed synthetic experiments and interesting real-world datasets.Some minor points:1 - I dind't find a definition of \lambda_{min}. I suppose it's the minimum eigenvalue (?) this should be made clear since notation overloads with \lambda for regularization parameter.2 - In the algorithm description, I presume you use OLS in line 11 (which is hinted at in the end of page 2). Please make this explicit in the algorithm description (you use "loss" to denote different losses in lines 5 and 11, and this causes confusion)3 - I think you never give details about *how* the dataset should be split. I feel it needs to be randomly (is that correct?). Could you please make this clear.4 - Second sentence of introduction starts with lower-case.Overall a really good paper.