Relating Leverage Scores and Density using Regularized Christoffel Functions
Statistical leverage scores emerged as a fundamental tool for matrix sketching and column sampling with applications to low rank approximation, regression, random feature learning and quadrature. Yet, the very nature of this quantity is barely understood. Borrowing ideas from the orthogonal polynomial literature, we introduce the regularized Christoffel function associated to a positive definite kernel. This uncovers a variational formulation for leverage scores for kernel methods and allows to elucidate their relationships with the chosen kernel as well as population density. Our main result quantitatively describes a decreasing relation between leverage score and population density for a broad class of kernels on Euclidean spaces. Numerical simulations support our findings.
*** UPDATE *** Thank you for your author feedback. I feel that the paper is strong, but discussion with fellow reviewers persuaded me to reduce my score from 9 to 8 due to their feedback that aspects of the presentation are unclear. ***  This paper makes a strong mathematical contribution to the concept of leverage, linking this to notions from the theory of reproducing kernels and, ultimately, proposing and studying (mathematically) a "regularised" Christoffel function. Such a contribution is, to my mind, timely and its theoretical emphasis is welcome at NIPS.   The paper reads well and the only comments that I have are extremely minor:  - for transpose in latex, ^T ought to be ^\top  - on p2 "which degree is" -> "whose degree is"  - on p4 "approximate integration" -> "approximates integration" and "we first precise" -> "we first make precise"  - on p4, technically L^\infty hasn't been defined.  - the overview of numerical cubature in methods in sec 2.4 was rather limited, and the authors could emphasise that in general a cubature rule should be selected based on properties of the function being integrated. For example, arbitrarily fast convergence for smooth integrands can sometimes be achieved with quasi-Monte Carlo methods.  - on p7 "numerical evidences" -> "numerical evidence"