Scalable Nonlinear Learning with Adaptive Polynomial Expansions
Can we effectively learn a nonlinear representation in time comparable to linear learning? We describe a new algorithm that explicitly and adaptively expands higher-order interaction features over base linear representations. The algorithm is designed for extreme computational efficiency, and an extensive experimental study shows that its computation/prediction tradeoff ability compares very favorably against strong baselines.
The paper proposes a supervised learning algorithm. It uses stochasticgradient descent and periodically expands the hypothesis space byintroducing new basis functions and adding corresponding components tothe weight vector. As such, as it processes more data, it fits morecomplex models.  The hypothesis space considered here are polynomialsand higher order monomials are gradually introduced to the model.The concept of growing the hypothesis space as more data is introducedis not new (training kernel methods with SGD exhibits this behavior),but in the proposed method, choosing which monomials to add to thehypothesis space is very cheap. The method selects monomials byinspecting the components of the weight vector instead of, say,selecting the monomial that would most reduce the training error (as amatching pursuit variant of the method might).The experiments are thorough and convincing. The method is nearly asfast as fitting a quadratic polynomial, yet just as accurate asfitting a cubic polynomial.The regret bound and its proof are a treat to read. It is pleasantthat the math works out so that more weight is placed on latercomparators.The authors might clarify that the regret bound offers no guidance onhow to choose the monomials to add. Nor does it offer guidance on whento expand the hypothesis space. Rather it states that any nestedhypothesis spaces of this kind (see my note about "this kind" below)do not have high regret.Some suggestions:1) The claim of Theorem 2 can be slightly generalized.  [.]_S(t) picksthe components of . in the support set S(t). But the proof goesthrough if [.]_S(t) is more generally a projection on a linearsubspace S(t). So Theorem 1 holds for any sequence of linear subspacesS(t) whose dimensionality expands with t.2) The two graphs of Figure 2 are informative. But the followingcombination of these two graph would have been more helpful to me: draw one curve peralgorithm as you currently do; let the x-axis be relative time (exactly as youalready compute it in Figure 2(a)), and let the y-axis be the number of datasetson which the algorithm obtains the best score among the other algorithms underthat running time.  [more detail: all of the algorithms being considered refine their solution over time. long below they satisfy their stopping criteria, each produces a useful result. i want to know how many seconds apple requires to beat its competition. i don't need to know its relative error, but rather whether it beats all of its competitors by time T. the graph would tell me how many datasets it dominates at time T.][in the interest of reproducible research, i am also upping my score by +1 because i noticed this technique is publicly available in VW now.]