Online and stochastic optimization methods such as SGD, ADAGRAD and ADAM are key algorithms in solving large-scale machine learning problems including deep learning. A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them. A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node. In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. Such a framework not only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions. We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings. Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.
This paper presents a consensus-based decentralized version of the Adam algorithm for online optimization. The authors consider an empirical risk minimization objective, which they split into different components, and propose running a separate online optimization algorithm for each component, with a consensus synchronization step that involves taking a linear combination of the parameters from each component before applying each component's individual parameter update. The final output is a simple average of the parameters from each component. 
The authors study the important problem of distributed optimization and focus on adapting existing state-of-the-art methods to this setting. The algorithm is clearly presented, and to the best of my knowledge, original. The fact that this work includes both theoretical guarantees for the convex and non-convex settings as well as numerical experiments strengthens the contribution.
On the other hand, I didn't find the actual method presented by the authors to be motivated very well. The main innovation with respect to the standard Adam/AMSGrad algorithm is the use of a mixing matrix W, but the authors do not discuss how the choice of this matrix influences the performance of the algorithm or how one should specify this input in practice. This seems like an important issue, especially since all of the bounds depend on the second singular value of this matrix. Moreover, arguments such as Corollary 10 do not actually imply that DADAM outperforms ADAM when this singular value is large, making it difficult to assess the impact of this work. The numerical experiments also do not test for the statistical significance of the results. 
There are also many typos that make the submission seem relatively unpolished.
Specific comments:
1. page 1: "note only". Typo.
2. page 2: "decentalized". Typo.
3. page 2: "\Pi_X[x]. If \Pi_X(x)...." Inconsistent notation.
4. page 3: "largest singular of matrix". Typo.
5. page 3: "x_t* = arg min_{x \in X} f_t(x)". f_t isn't defined in up to this point.
6. page 4: "network cost is then given by f_t(x) = \frac{1}{n} \sum_{i=1}^n f_{i,t}(x)" Should the cost be  \frac{1}{n} \sum_{i=1}^n f_{i,t}(x_{i,t})? That would be more consistent with the definition of regret presented in Reg_T^C. 
7. page 4: "assdessed". Typo.
8. page 4: " Reg_T^C := \frac{1}{n} \sum_{i=1}^n \sum)_{t=1}^T f_t(x_{i,t})..." Why is this f_t and not f_{i,t}?
9. page 4: "\hat{v}_{i,t} = v_3 ..." You should reference how this assignment in the algorithm relates to the AMSGrad algorithm. Moreover, you should explain why you chose to use a convex combination in the assignment instead of just the max.
10. page 5: Definition 1. This calculation should be derived and presented somewhere (e.g. in the appendix).
11. page 5: Assumption 3. The notation for the stochastic gradient is not very clear and easily distinguishable from the notation for the deterministic gradient.
12. page 5: Theorem 4. D_T can be very large in the bound, which would make the upper bound meaningless. Can you set hyperparameters in such a way to minimize it? Also, what is the typical size of \sigma_2(W) that one would incur?
13. page 6: Remark 6. This remark seems misleading. It ignores the log(T) and D_T terms, both of which may dominate the data dependent arguments.
14. page 6: "The update rules \tilde{v}_{i,t}...". \tilde{v}_{i,t} is introduced but never defined.
15. page 6: Last display equation. The first inequality seems like it can be an equality.
16. page 7: Equation (14). Doesn't the presence of \sigma_2(W) imply that the O(1/T) term may not be negligible? It would also be helpful to give some examples of how large T needs to be in (15a) and (15b) in order for this statement to take effect.
17. page8: "distributed federated averaging SGD (FedAvg)". What is the reference for this? It should be included here. It should probably also be mentioned in the introduction as related work.
18. page 9: Figure 1. Without error bars, it is impossible to tell the statistical significance of these results. Moreover, how sensitive are these results to different choices of hyperparameters?
19. page 9: "obtain p coefficients". What is p in these experiments?
20. page 9: Metropolis constant edge weight matrix W". What is \sigma_2(W) in this case?
21. page 10: Acknowledgements. This shouldn't be included in the submission.