Extending Gossip Algorithms to Distributed Estimation of U -Statistics
Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of Ustatistics, relying on more expensive averaging over pairs of observations, is a less investigated area. Yet, such data functionals are essential to describe global properties of a statistical population, with important examples including Area Under the Curve, empirical variance, Gini mean difference and within-cluster point scatter. This paper proposes new synchronous and asynchronous randomized gossip algorithms which simultaneously propagate data across the network and maintain local estimates of the U -statistic of interest. We establish convergence rate bounds of O(1/t) and O(log t/t) for the synchronous and asynchronous cases respectively, where t is the number of iterations, with explicit data and network dependent terms. Beyond favorable comparisons in terms of rate analysis, numerical experiments provide empirical evidence the proposed algorithms surpasses the previously introduced approach.
This paper is about computing higher-order statistics measured over pairs of observations, where the observations themselves are distributed across a network of nodes. These U-statistics include many quantities of interest such as the sample variance, correlation and scatter coefficients, and other aggregate measures. The main goal of the paper is to propose a gossip-style decentralized aggregation/averaging algorithm to compute these statistics.Overall, this is a nice contribution to the literature on gossip algorithms, but the authors should make a better connection to existing approaches and ideas.This is a "heavy" review. I did not have time to verify the correctness of the material in the supplement.Comments1) The main trick on the algorithm side is that when two nodes exchange their estimate values and average them, they also swap a secondary state variable representing their initial observation. This means that the initial observations are each performing a random walk (albeit coupled). Mentioning this more clearly in the main manuscript would help and also would motivate the analysis method via the multiple copies of the graph.2) There has been a lot of work on gossip approaches for computing statistics beyond the references cited by the authors. A snapshot of the state of the art from a few years ago is the Dimakis et al. Proc. IEEE paper. Situating the current work in the context of the works mentioned there might be more helpful than the somewhat cursory explanation in the current manuscript.3) On page 4 it might be worth clarifying what the induced distribution on edges is from the "node wakes up, contacts neighbor" process. The phrase "drawing a random edge of the network" does not specify the distribution.4) There is a natural other approach to this problem, which is to allow more memory at each node and essentially use the same contact process to have each node flood its own value across the network. Nodes store the received observations (tagged with the node who initially possessed it). At each time nodes can calculate the U statistic on the subset of the data that they possess. How would this compare to the current method? This may help show why the averaging can be helpful, or show the loss in performance when the memory at each node is limited. Without this comparison it's not clear that the rate 1/t is good or bad. For expanders the flooding method may be quite good.5) The authors should take a look at the work on lifting constructions for accelerating gossip algorithms, as it seems quite related to their construction in the paper. See for example:K. Jung, D. Shah, and J. Shin, "Fast gossip through lifted Markov chains," in Proc. Allerton Conf. on Comm., Control, and Comp., Urbana-Champaign, IL, Sep. 2007. W. Li and H. Dai, "Location-aided fast distributed consensus," in IEEE Transactions on Information Theory 56(12), 2010. F. Chen, L. Lovasz, and I. Pak, "Lifting markov chains to speed up mixing," in Proceedings of the thirty-first annual ACM symposium on Theory of computing. ACM, 1999, pp. 275-281.6) Analysis of gossip algorithms tend to make a worst-case assumption on the initial data distribution. That is, the guarantee is over all initial data vectors $X$. This is not clear in the problem statement of the statement of the theorem. What is the assumption on the data distribution and what are the guarantees on the error as a function of the initial data?Small things - p1: The bipartite definition using (S \times T) \cup (T \times S) implies directed graphs. - p4: "let us stress out that" --> "we stress that" - p4: "awakes and contact" -- > "awakes and contacts" - p6: "requires to initiate" --> "requires initiating" - p7: "allows to create" --> "allows us to create"AFTER AUTHOR FEEDBACK: I think the clarifications and modifications suggested by the authors are good. I am revising my initial score of 7 up to an 8.I think that they should still take a close look at the lifting constructions (esp. LADA) and modify the related work.Furthermore, the experiments should include the baseline comparisons.