Coresets for Scalable Bayesian Logistic Regression
The use of Bayesian methods in large-scale data settings is attractive because of the rich hierarchical models, uncertainty quantification, and prior specification they provide. Standard Bayesian inference algorithms are computationally expensive, however, making their direct application to large datasets difficult or infeasible. Recent work on scaling Bayesian inference has focused on modifying the underlying algorithms to, for example, use only a random data subsample at each iteration. We leverage the insight that data is often redundant to instead obtain a weighted subset of the data (called a coreset) that is much smaller than the original dataset. We can then use this small coreset in any number of existing posterior inference algorithms without modification. In this paper, we develop an efficient coreset construction algorithm for Bayesian logistic regression models. We provide theoretical guarantees on the size and approximation quality of the coreset -both for fixed, known datasets, and in expectation for a wide class of data generative models. Crucially, the proposed approach also permits efficient construction of the coreset in both streaming and parallel settings, with minimal additional effort. We demonstrate the efficacy of our approach on a number of synthetic and real-world datasets, and find that, in practice, the size of the coreset is independent of the original dataset size. Furthermore, constructing the coreset takes a negligible amount of time compared to that required to run MCMC on it.
This paper aims to develop a method to construct a coreset for Bayesian logistic regression. A coreset is a weighted subset of data, which is hopefully much smaller than the original dataset, and which yields provably similar results when existing inference algorithms are applied to it.  Intuitively, this procedure leverages a k-clustering of the data (which must first be constructed) to compute a weight for each data point and to compute the total size of the coreset. The coreset is then resampled from the initial dataset using the computed weights.  This paper restricts its focus to the Bayesian logistic regression model only.  Furthermore, the theory restricts the parameter space to be within a Euclidean ball of a pre-specified radius R (as opposed to unbounded R^D space of parameters, which is the typical setting for logistic regression). The authors also provide negative theoretical results showing that their theoretical framework would not work on an unbounded parameter space, to justify the bounded parameter assumption.  Empirically, the authors provide empirical results on three datasets, which aim to show that inference on coresets performs better than on random subsampling of the data, and provides a good approximation to inference on the full data when the coreset size grows large enough.
* I have one major gripe with the practicality of using coresets versus recent scalable inference strategies, such as minibatch inference methods.  Note that in minibatch inference methods, at each iteration, a small subset of the data is sampled from the full dataset and used to make an update; these methods take advantage of the redundancy in data to perform inexpensive updates. In this paper, coresets reduce the total dataset size by, in some sense, approximating the dataset with a smaller group of (weighted) examples. However, when coresets are used in existing inference algorithms (such as these minibatch algorithms), it seems to me that a very similar procedure will occur: a small subset of this approximate, weighted dataset will be drawn, and used to make an update. I am not convinced this would actually speed up inference (i.e. decrease the number of iterations needed or the time of each iteration), and I don't see a great justification given in this paper. In a sense, I feel that the main thing happening here is that the data is approximated in a smaller/compressed fashion; I can see how this might help with data storage concerns, but I don't see a great justification for why it would appreciably speed inference over existing minibatch methods (especially considering a coreset must be constructed before inference can proceed, which adds additional inference time to this method). One way to demonstrate this would be with timing comparison plots that explicitly show that coresets yield faster inferences given large datasets when compared to minibatch methods---however, no direct experiments of this sort are given.  To summarize this complaint, I feel there is some disconnect between what this paper showed and what it set out to do: we know that smaller datasets typically allow for faster inference, and we know that coresets produces smaller (weighted) datasets. However, it’s not convincingly shown that when coresets are used with (or in comparison with) recent efficient methods, they actually allow for faster inference.   * After the coreset is constructed, the authors intend for existing inference algorithms to be applied to the coreset. However, it is not immediately clear to me how a weighted set of data can be directly used with existing inference algorithms. One could duplicate or resample data points based on their weight, and use this new dataset directly in existing inference algorithms --- is this what happens? Or is there a general-purpose way to use the coreset weights in existing inference algorithms?  * Regarding the definition of an \epsilon-coreset: on page 3, an \epsilon-coreset is defined to be a coreset in which the difference between the log marginal likelihoods of the coreset and the full data are bounded.  Intuitively, I feel the most straightforward goal is to design a coreset in which the approximate coreset posterior is similar to the posterior of the full dataset (i.e. the coreset posterior and true posterior are bounded). However, I have no intuition about how bounding the difference between the log marginal likelihoods corresponds to this goal. Perhaps these two bounds do directly correspond in some way; if so, I feel the paper should do more to explicity state how well an \epsilon-coreset posterior approximates the true posterior. Or at least provide more justification about why bounding the log marginal likelihoods is the appropriate goal.  * Regarding empirical results: on page 7, the main body of empirical results show that a big enough subset eventually yields a good approximation to the test log-likelihood. However, as I describe above in more detail, the results do not seem to focus on what I feel is the direct goal of this work (which is to allow for faster inference in Bayesian logistic regression by first constructing a coreset). Results of this sort would aim to show that a coreset can be constructed, and inference on this coreset completed, in a faster time than it would take to perform inference on the full original dataset. I feel that results of this sort are particularly important since there exists many popular minibatch algorithms developed in recent times, in which each sample can typically be generated very quickly. It would be nice to see if running these minibatch inference algorithms on a coreset versus the entire dataset has a meaningful speedup.  * Experimental comparisons: there were not many experimental comparisons. The main experimental comparison seems to be random subsampling --- it slightly confused me that (I don't believe) this comparison was explicitly described in the paper. However, I assume this is just inference run on a random (uniformly drawn) subset of data of size M.  * What is a good way to determine the parameter space radius R (which is required as an input to the coreset algorithm)? The authors mention that restricting the parameter space to a Euclidean ball is justified since the data is usually preprocessed (to be mean zero and variance 1), which causes each parameter component to be small (e.g. absolute value less than 4) --- however, this statement is not rigorously justified, and I am not sure if it is always true. Is there any theoretically justified way to choose a correct R?  * It seems like the coreset procedure described in this paper is mainly dependent on how well the data can be approximated by a set of clusters.  Theoretically speaking, is there any intuition about a good way to choose the number of clusters, and is the procedure dependent upon this choice?  I do, however, notice that the choice of k over a range is demonstrated empirically.