Empirical Risk Minimization in Non-interactive Local Differential Privacy Revisited *
In this paper, we revisit the Empirical Risk Minimization problem in the noninteractive local model of differential privacy. In the case of constant or low dimensions (p n), we first show that if the loss function is (∞, T )-smooth, we can avoid a dependence of the sample complexity, to achieve error α, on the exponential of the dimensionality p with base 1/α (i.e., α −p ), which answers a question in [19]. Our approach is based on polynomial approximation. Then, we propose player-efficient algorithms with 1-bit communication complexity and O(1) computation cost for each player. The error bound is asymptotically the same as the original one. With some additional assumptions, we also give an efficient algorithm for the server. In the case of high dimensions (n p), we show that if the loss function is a convex generalized linear function, the error can be bounded by using the Gaussian width of the constrained set, instead of p, which improves the one in [19].
- Summary:   This paper considers the problem of empirical risk minimization in the non-interactive setting of the local model of differential privacy. In this setting, each user (holding one data point) is required to send a differentially private signal to the server without any prior interaction with the server or other users. Then, the server collects the users' signals and uses them to solve the ERM problem. The most relevant previous work is [19] that shows that any protocol that is based on first (or second) order methods (e.g., gradient descent and other variants) must require sample size \Omega(\alpha^{-p}) if it were to achieve error \alpha (where p is the dimensionality of the parameter space). This reference also gives upper bounds of the same order for non-interactive ERM under Local Differential Privacy (LDP) for the class of Lipschitz loss functions and the class of Lipschitz, convex loss functions.   This paper revisits this problem under some smoothness assumptions on the loss function, and devises new algorithms for this problem based on polynomial approximation techniques. The goal is to improve the previous bounds for non-interactive ERM under LDP for some classes of smooth loss functions.   When the sample size (n) < the dimensionality (p): despite success in showing improved sample complexity bounds, the resulting bounds still exhibit exponential dependence on p. For a sub-class of smooth functions (with bounded derivatives up to some constant order), the resulting bound is still ~ \alpha(-\Omega(p)), albeit with a better exponent than the one in [19] devised for Lipschitz loss functions. For a more restrictive class of smooth functions (with bounded derivative of any order), the resulting bound shows a better dependence on \alpha (namely, \alpha^{-4}), yet it still has from an extra factor that is exponential in p (about 4^{p^2}).   When p > n: the paper gives an algorithm that results in sample complexity bounds that only depend (albeit exponentially) on the Gaussian width of the parameter set and n (when the loss function is a generalized linear function).   The paper also shows improved constructions that has better running time and communication complexity than the ones in [19].   - Strengths:   1- The paper studies an important problem. The work is well-motivated. The introduction, as well as the outline of the paper and the high-level description of the algorithms and technical contributions, are nicely written. However, there are some parts in the technical sections that are not as clear (e.g., undefined notation, missing definitions), but these were not very critical (also these issues are understandable considering the space limitations).  2- The paper gives new algorithms for non-interactive ERM under LDP based on techniques from polynomial approximation. The paper also shows improved constructions in terms of running time and communication complexity based on some ideas from [2]. (however, there are some issues need to be addressed and clarified further (See "Weaknesses" below)).   3- Rigorous and original analyses.   - Weaknesses:   1- Despite the strengths of the paper, the results are not quite impressive. The improvements in the bounds are hardly any useful even for p = 5, e.g., a sample complexity bound that scales with 4^{p^2} \alpha^{-4} is still not meaningful in this case (regardless of the fact that it's better than \alpha^{-2p} for some range of \alpha and p).   2- The paper raises an interesting question in the introduction and uses it to build up part of the motivation for this work, but it does not seem to address at all. The paper notes that in some special cases of loss functions (e.g., squared loss [the linear regression case in [19]] and logistic loss [24]) one can have bounds that are free from this exponential dependence on p in the non-interactive setting, and continues to say "this propels us the following natural questions .... Is it possible to introduce some natural conditions on the loss function that guarantee non-interactive _x000f_-LDP with sample complexity that is not exponential in the dimensionality p?". Although the class of smooth functions considered in this paper contains the two aforementioned examples of loss functions, there is no indication that the derived bounds can ever be free from this exponential dependence on p for those special cases of loss functions.   3- The bounds derived for generalized linear functions in terms of Gaussian width (for p > n case) are also hardly practical. Even if the Gaussian width is as small as log(p), the error bounds still scale roughly as ~ (1/n)^{1/\Omega(n)} (Corollary 3). Comparing this to known non-private nearly dimension-independent risk bounds for generalized linear models, it's not clear if the derived bounds are any useful.   4- There is an unquantified dependence on p in Corollary 2, Theorems 4 and 5 via the parameter D_p. The paper introduces D_p (once in the prelim section) as a "universal constant" that depends only on p. But, I don't see how it's a constant if it depends on p? Can this "constant" be at least upper and lower bounded by some truly universal constants independent of p? I think this dependence needs to be explained further. As stated, it's not clear if this D_p does not add an extra factor to the bound making it worse.   5- Some of the discussion in the paragraph after Corollary 2 is actually not clear. Why is it when h --> \infty, the dependence on the error becomes \alpha^{-2}? Isn't it alpha^{-4}?  6- In the analysis of Algorithm 3, the issue of shared/public randomness required in the setup needs to be further discussed.   ------------------------------- -------------------------------    ****Post Rebuttal Comments:  -----------------------------------  After reading the authors' rebuttal, I am inclined to keep my score as is. The authors agreed with several points made in my review, and they clarified most of the issues I had questions about. I think the work is theoretically interesting and removing the 1/\alpha from the base of the exponential dependence is nice. However, I still question the extent of the practicality of the main results especially with the prohibitive exponential dependence on the square of the dimensions in the final bounds. Also, the paper could not provide any insight for the reason why there are non-interactive schemes for some special cases of smooth losses (e.g, linear regression) that do not suffer from this exponential dependence at all.