Derivative Estimation in Random Design
We propose a nonparametric derivative estimation method for random design without having to estimate the regression function. The method is based on a variance-reducing linear combination of symmetric difference quotients. First, we discuss the special case of uniform random design and establish the estimator's asymptotic properties. Secondly, we generalize these results for any distribution of the dependent variable and compare the proposed estimator with popular estimators for derivative estimation such as local polynomial regression and smoothing splines.
Paper 1768  This paper considers the univariate regression setting Y_i = r(X_i) + epsilon_i for X_i taking value in [0,1] and independent epsilon_i. The authors study the estimation of the derivative r'(x) of the regression function for x in the interior (0,1). The authors use a weighted finite difference ratio estimator and analyze its behavior in the random design setting where X_i has the uniform density. The main result yields a pointwise rate of convergence for the estimator on the data points X_2, ..., X_{n-1}. Extensions to non-uniform design points are discussed.   I find the problem and the results interesting. I think the paper is close to a state where it can be published but it needs to be developed some more before it is ready.  I like the results but I think they are a bit limited in scope and leave too many immediate natural questions unanswered. The main result characterizes the bias and variance of the estimated derivative at a data point X_i; it is natural to think about the behavior of the estimator at an arbitrary point x in the interior of of (0,1). I would guess that the estimator still behaves well under the same bounded second derivative condition that the authors imposes on the regression function r.  Another natural question is whether the estimated derivative is asymptotically normal. Theory aside, the expression (3) immediately leads one to think about kernel weighing. I hypothesize that using a kernel weighted version of (3) gives an estimator whose behavior is similar to that of one proposed by the authors. Finally, in the case where X is non-uniform but has a density that is bounded away from 0, I would imagine that the exact same method should work with the same, up to a constant, bounds on the bias and variance. It seems excessive in the case of non-uniform X to follow what the authors suggests in section 2.7, which involves estimating the density itself.  I understand it is unfair for a reviewer to speak on and on about what the authors could do. I raise these open questions because I think they are natural and not too difficult to address. I could be wrong of course, in which case, I would be happy if the authors could discuss why they are difficult.   A second issue I have with the paper, which is not a huge issue, is that the mathematical statements are loose. Some small degree of mathematical rigor is needed for a NIPS statistical methodology paper and this paper does not quite reach that standard. In the statement of theorem 1, it does not make sense to condition on the samples U because that yields the fixed design setting. I think what the authors intend is E_U[ \hat{Y}_i - r'(U_i) | U_i], that is, the expected estimate at the point U_i fixing only the value of the point U_i. I think it is clearer to have an unconditional statement, which holds by just taking an iterated expectation. I list a few other issues.  + In expression (4), it is very strange to have the subscript on Y be just ``i'' and to have the subscript on U be ``(i)''. I suggest saying something like, ``by reindexing if necessary, assume that U_1 \leq U_2 \leq ... ``.  + In Section 1.1, it is not technically correct to write m(x) = E(Y) and m(X) = E[Y | X]. The correct expression is m(x_i) = E[Y_i] in the first case and m(x) = E[ Y | X = x] in the second case.  + The authors say that ``we require P(U_(l) = U_(m)) = 0'', but that is already true since the U's are uniformly distributed on [0,1]. I suggest saying something like, ``we observe that because U_i has a density, P(U_(l) = U(m)) = 0 and thus the estimator is well-defined on U_i for all i = 2, ..., n-1''.   + I recommend giving an explicit rate in corollary 1, with k chosen optimally.   + Corollary 2 isn't quite right because the the first term is based on an upper bound of the bias.   + The authors say in line 202 that they need the existence of a density in order for expression (10) to hold but they in fact need the existence of a density earlier. The distribution of X_1 needs to be absolutely continuous with respect to the Lebesgue measure in order for F(X_1) to be uniformly distributed. If the distribution of X_1 has a component singular to the Lebesgue measure, then the quantile function is not the inverse of the distribution function F and F(X_1) is not uniformly distributed.   + I don't know of a result in [1] that describes the optimality of derivative estimators. [1] mostly deals with the estimation of an additive regression function itself. It would be helpful if the authors could provide a reference to the particular theorem in the paper.  [1] C. Stone. Additive regression and other nonparametric models.   EDIT: I have read the authors' response. I agree that the conditional on U statement as it appears in the paper is actually correct; I had misunderstood it before. I also agree that F needs to only be continuous and not absolutely continuous as I had mistakenly stated before. I apologize for these mistakes.   I do not have a reference for using kernel weighing. It just strikes me that the authors' weighing scheme resembles kernel weighing where the kernel is K(u) = 1/u^2 * I(u \leq 1). See an analysis of this kernel in [1]. Bandwidth selection is certainly an issue but so is the selection of "k" in the authors' scheme.   I still do not understand why estimating the density f(X) is necessary. It seems that Lemma 1 would hold if X is non-uniform but has a density that is bounded away from 0. It would be great if the authors could comment on this in the paper.   I don't think that studying the behavior of \hat{r} is "sufficient for another paper", especially since the authors already use the random design setting. I think not being able to say anything about the behavior of \hat{r} except on the sample points is the most significant weakness of the paper.   I have upgraded my assessment of the paper though I still recommend that the authors take some time to further develop the paper before submitting it for publication. But, I don't have very strong objections to accepting the paper as it is.   [1] Belkin, M, Rakhlin, A, and Tsybakov, A. (2018) Does data interpolation contradict statistical optimality?  Arxiv Preprint: 1806.09471