Many machine learning problems reduce to the problem of minimizing an expected risk. Incremental gradient (IG) methods, such as stochastic gradient descent and its variants, have been successfully used to train the largest of machine learning models. IG methods, however, are in general slow to converge and sensitive to stepsize choices. Therefore, much work has focused on speeding them up by reducing the variance of the estimated gradient or choosing better stepsizes. An alternative strategy would be to select a carefully chosen subset of training data, train only on that subset, and hence speed up optimization. However, it remains an open question how to achieve this, both theoretically as well as practically, while not compromising on the quality of the final model. Here we develop CRAIG, a method for selecting a weighted subset (or coreset) of training data in order to speed up IG methods. We prove that by greedily selecting a subset S of training data that minimizes the upper-bound on the estimation error of the full gradient, running IG on this subset will converge to the (near)optimal solution in the same number of epochs as running IG on the full data. But because at each epoch the gradients are computed only on the subset S, we obtain a speedup that is inversely proportional to the size of S. Our subset selection algorithm is fully general and can be applied to most IG methods. We further demonstrate practical effectiveness of our algorithm, CRAIG, through an extensive set of experiments on several applications, including logistic regression and deep neural networks. Experiments show that CRAIG, while achieving practically the same loss, speeds up IG methods by up to 10x for convex and 3x for non-convex (deep learning) problems.
This paper proposes a novel extension to SGD/incremental gradient methods called CRAIG. The algorithm selects a subset of datapoints to approximate the training loss at the beginning of each epoch in order to reduce the total amount of time necessary to solve the empirical risk minimization problem. In particular, the algorithm formulates a submodular optimization problem based on the intuition that the gradient of the problem on the selected subset approximates the gradient of the true training loss up to some tolerance. Each datapoint in the subset is a medoid and assigned a weight corresponding to the number of datapoints in the full set that are assigned to that particular datapoint. A greedy algorithm is employed to approximately solve the subproblem. Theory is proven based on based on an incremental subgradient method with errors. Experiments demonstrate significant savings in time for training both logistic regression and small neural networks.
Strengths:
The proposed idea is novel and intriguing, utilizing tools from combinatorial optimization to select an appropriate subset for approximating the training loss. Based on the experiments provided in the paper, it does appear to yield a significant speedup in training time. It is interesting to observe how the order of the datapoints matter significantly for training, and that CRAIG is also able to naturally define a good ordering of the datapoints for SG training. This is strong algorithmic work.
Weaknesses:
Some questions I had about the work:
- How well does one have to approximate $d_{ij}$ in order for the method to be effective? The authors provide an approach to approximate this for both logistic regression and neural networks. How does one guarantee that one is obtaining the maximum over $x in \mathcal{X}$ for neural networks via backpropagating only on the last layer? Does taking this maximum matter?
- How does one choose $\epsilon$? Is this related to how $d_{ij}$’s are approximated?
- If one were to consider an algorithm that samples points from this new distribution over the data given by CRAIG, if one were to include the weight $\gamma_j$ into the algorithm, would the sample gradient be unbiased? What if one were to simply use $\gamma_j$ to weight that particular sample in the new distribution?
- In machine learning, the empirical risk (finite-sum) minimization problem is an approximation to the true expected risk minimization problem. What is the effect of CRAIG on the expected risk? Is there any deterioration in generalization performance?
- In page 4, what does the $\min_{S \subseteq V}$ refer to? Should the equation be interpreted as with the set $S$ fixed or not?
- Theorems 1 and 2 are stated a bit non-rigorously. Are these theorems for fixed $k$? What does it mean for these bounds that $k \rightarrow \infty$?
- In Theorems 1 and 2, what is the bound on the steplength in order to obtain the convergence result for $\tau = 0$?
- In Theorem 1 for $0 < \tau < 1$, why does one obtain a result where $\|x_k – x_*\|^2 \leq 2 \epsilon R / \mu$, why is the distance to the solution bounded by a constant? What if one were to initialize $x_0$ to be such that $\|x_0 – x_*\|^2 > 2 \epsilon R / \mu$? (Similar for Theorem 2.)
- In the experiments, how is the steplength and other hyperparameters tuned? Are multiple trials run? 
- Is $\epsilon$ used to determine the subset or is it based on a predetermined subset size?
- How do the final test losses compare between CRAIG and the original algorithms?
- How do the relative distances (rather than the absolute distance) to the solution behave?
- How does CRAIG perform over multiple epochs? How does the algorithm transition when the subset is changed (as in neural networks)?
- Why does CovType appear more stable with the shuffled version over the other datasets? Is the stability related to the distribution of the weights $\gamma_j$?
Some grammatical errors/typos/formatting issues:
- Equation (9) needs more space between the $\forall x, i, j$ and the rest of the equation.
- What is $\Delta$ on page 5? Is it supposed to be $F$?
- On page 8, And should not be capitalized.
- Page 14, prove not proof
- Page 14, subtracting not subtracking
- Page 16, cycle not cycke
Overall, although I like the ideas in the paper, the paper still needs some significant amount of refining in terms of both writing and theory, as well as some additional experiments to be convincing. If the comments I made above were addressed, I would be open to changing my decision.