On Communication Cost of Distributed Statistical Estimation and Dimensionality
We explore the connection between dimensionality and communication cost in distributed learning problems. Specifically we study the problem of estimating the mean✓ of an unknown d dimensional gaussian distribution in the distributed setting. In this problem, the samples from the unknown distribution are distributed among m different machines. The goal is to estimate the mean✓ at the optimal minimax rate while communicating as few bits as possible. We show that in this setting, the communication cost scales linearly in the number of dimensions i.e. one needs to deal with different dimensions individually. Applying this result to previous lower bounds for one dimension in the interactive setting [1] and to our improved bounds for the simultaneous setting, we prove new lower bounds of ⌦(md/ log(m)) and ⌦(md) for the bits of communication needed to achieve the minimax squared loss, in the interactive and simultaneous settings respectively. To complement, we also demonstrate an interactive protocol achieving the minimax squared loss with O(md) bits of communication, which improves upon the simple simultaneous protocol by a logarithmic factor. Given the strong lower bounds in the general setting, we initiate the study of the distributed parameter estimation problems with structured parameters. Specifically, when the parameter is promised to be s-sparse, we show a simple thresholding based protocol that achieves the same squared loss while saving a d/s factor of communication. We conjecture that the tradeoff between communication and squared loss demonstrated by this protocol is essentially optimal up to logarithmic factor.
This paper explores the connection between dimensionality and communication cost in distributed learning problems. In this problem, the samples from the unknown distribution are distributed among m different machines. The goal is to estimate the mean  at the optimal minimax rate while communicating as few bits as possible. This paper shows that in this setting, the communication cost scales linearly in the number of dimensions. Applying this result to previous lower bounds for one dimension in the interactive setting [4] and to the improved bounds for the simultaneous setting, this paper proves new lower bounds of \Omega(md/ log(m)) and \Omega(md) for the bits of communication needed to achieve the minimax squared loss, in the interactive and simultaneous settings respectively. Besides, this paper demonstrates an interactive protocol achieving the minimax squared loss with O(md) bits of communication. Given the strong lower bounds in the general setting, this paper initiates the study of the distributed parameter estimation problems with structured parameters. Specifically, when the parameter is known to be s-sparse, it  shows a protocol achieving the minimax squared loss with high probability and with communication cost proportional to s rather than the dimension d of the ambient space.Quality: The authors need to assume that the readers have no knowledge on the problem in [4].Clarity:  In Section 2, definitions are missing for several notions:0. What is the definition of s-sparse?1. In the definition of R(\hat{\theta},\theta), why the expectaion is over \hat{\theta},X,\theta? How is  the mean \vec{\theta}(Y) of Y obtained? Why is Y in {\cal X}^n? Is there any example for transcript?2. I cannot follow below Private/Public Randomness. For example,a. What is private and public randomness?  What is a protocol? Give definitions to them. b. Why can the public randomness be shared among the machines before the start of the protocol?c. Why does the protocol work well on average over all public randomness?d. Why can the machine use private randomness to hide information from other machines in a protocol?e. The definition ``(\Pi,\vec{\theta}) solves T(d,m,n,\sigma^2,{\cal D}_\theta^d) with C and R.Oiginality: I believe that new lower and upper bounds are derived for the interactive and simultaneous settings, and that the negative result is obtained. It is fair that the authors explain how difficult the derivation fom [4] is.Significance: I do not understand why ''the communication cost scales linearly in the number of dimensions" is a news. In fact, ituitively, this seems to be obvious while they mathematically derived the property. The authors need to explain why the result is different from what they expected.