On the Fine-Grained Complexity of Empirical Risk Minimization: Kernel Methods and Neural Networks
Empirical risk minimization (ERM) is ubiquitous in machine learning and underlies most supervised learning methods. While there is a large body of work on algorithms for various ERM problems, the exact computational complexity of ERM is still not understood. We address this issue for multiple popular ERM problems including kernel SVMs, kernel ridge regression, and training the final layer of a neural network. In particular, we give conditional hardness results for these problems based on complexity-theoretic assumptions such as the Strong Exponential Time Hypothesis. Under these assumptions, we show that there are no algorithms that solve the aforementioned ERM problems to high accuracy in sub-quadratic time. We also give similar hardness results for computing the gradient of the empirical loss, which is the main computational burden in many non-convex learning tasks.
The paper make use of (relatively) recent advances in complexity theory to show that of many common learning problems do not allow subquadratic time learning algorithms (given the veracity of the "Strong Exponential Time Hypothesis").
I appreciate that the authors do not oversell their results: They clearly state that they provide a worst-case analysis. Also, the results are not surprising. For instance, finding the exact solution of any kernel method requires the computation of the full kernel matrix, which is already quadratic in number of training examples. Reducing this computation time would imply that one can compute an approximation of the exact solution without computing the full kernel matrix, which is intuitively unlikely, unless he makes extra assumptions on the problem structure (i.e., the nature of the data-generating distribution). Hence, I would like the authors to comment on whether or not similar results could be obtained for the linear counterparts of studied ERM algorithms.
That being said, I consider that this is an important contribution. Up to my knowledge, this work is significantly different than common statistical learning theory papers. It may bring fresh air to our community, by bringing tools and insights from the complexity theory literature. From my point of view, the reductions from a discrete optimization problem to a continuous one are clever.
It is also worth noting that the paper is written rigorously (more than the average NIPS paper, I would say). Despite the hardness of the subject, it is easy to get the main ideas, notably the reductions from one problem to the other.