-regression with Heavy-tailed Distributions
In this paper, we consider the problem of linear regression with heavy-tailed distributions. Different from previous studies that use the squared loss to measure the performance, we choose the absolute loss, which is capable of estimating the conditional median. To address the challenge that both the input and output could be heavy-tailed, we propose a truncated minimization problem, and demonstrate that it enjoys an O( d/n) excess risk, where d is the dimensionality and n is the number of samples. Compared with traditional work on 1 -regression, the main advantage of our result is that we achieve a high-probability risk bound without exponential moment conditions on the input and output. Furthermore, if the input is bounded, we show that the classical empirical risk minimization is competent for 1 -regression even when the output is heavy-tailed.
In this manuscript the authors consider the problem of linear regression in the case where the distributions of the input and output are heavy-tailed. The goal of this task is to find a linear function that is gives the best description for transforming the input into the observed output. Here the term best means that this function should minimize the risk, which is the expected difference between the function applied to the input and the observed output under a specific loss function. Although in many standard work on linear regression the squared loss is considered, the authors argue that for heavy-tailed in- and output the absolute loss is more suited.  To approach the linear regression problem the authors use a truncation method, which finds a linear function that minimizes a rescaled empirical average of a so-called truncation function. The difference between the risk of the true linear regression and that of the truncation method is called the excess risk. The main results of this manuscript provide bounds on the excess risk, under certain conditions on the input distribution. In particular, using epsilon nets, they obtain an improved excess risk bound for linear regression with absolute loss, which they further improve using additional assumptions on the input data. The overall setup of the manuscript is clear. The authors explain the problem of linear regression and comment on different approaches and techniques in the literature, especially related to heavy-tailed distributions. The proofs are easy to follow and I did not find any errors.  The problem of linear regression with heavy-tailed distributions for input and even output data is relevant since many interesting systems in the real-world exhibit such behaviors. The innovative aspect of this work is the combination of absolute loss linear regression and truncation methods for a wide class of truncation functions. My main concern with this work is that although the title and many parts of the text refer to heavy-tailed distributions, these are never defined. I understand that one has an intuitive idea of what heavy-tailed means, but these are well defined objects, see for instance Definition 2.2 or Theorem 2.6 in Foss et al. An introduction to Heavy-tailed and subexponential distributions. The reason I bring this up is that nowhere in any of the proofs could I find any argument that used properties of heavy-tailed distributions. All results in this manuscript are proven under the three main assumptions. It is also worth noting that often real-world data is heavy-tailed with diverging second moment, which violates Assumption 2. Moreover, it is not clear at all if one should expect Assumption 3 to hold for simple heavy-tailed distributions such as Pareto or Log-normal. This is crucial since if this is not the case then it would make the results stated here much less useful.  To summarize, the paper is well-written and addresses a relevant problem by making a novel combination of absolute loss linear regression with truncation methods. The proofs are easy to read and no errors were found in them. However, the manuscript does not make use of any properties of heavy-tailed distributions and does not even discuss the validity of its assumptions for some standard examples of such distributions. Therefore, I would only weakly recommend this paper for acceptance to NIPS 2018. ADDED AFTER FEEDBACK: The authors addressed my questions and concerns regarding the use of the term heavy-tailed and the existence of second moments. Although I personally still believe that working with heavy-tailed (more specifically regularly-varying) distributions in the setting of infinite second moment seems more realistic, I see the authors point of view and am fine with the setting used in this paper. My other small questions where also addressed properly and hence I stand by my previous decision on this paper. Below is a small list of additional comments. Main text: Line 29, [heavy-tailed]: Please provide a definition of this. Equation (2): Why do you need the \frac{x^2}{2} term inside the function. It seems that this forces the finite second moment condition? Could you elaborate a bit on this? Line 77: Here you seem to indicate that a paper from 2011 extends a result from 2012. I did not find any information that the 2012 was available before the one from 2011. Could you maybe rephrase this sentence. Assumption 2: How realistic is the finite second moment assumption with respect to real-world data? Especially in the heavy-tailed regime this might be false most of the times. Could you discuss this a bit? Line 143-144: I do not understand why you would not simply use this relaxed assumption. Could you explain this? Line 201, [the omitted ones…]: What does ones refer to here. This is not really clear from the text. Are they lemmas or equations?