Federated learning, where a global model is trained by iterative parameter averaging of locally-computed updates, is a promising approach for distributed training of deep networks; it provides high communication-efficiency and privacypreservability, which allows to fit well into decentralized data environments, e.g., mobile-cloud ecosystems. However, despite the advantages, the federated learning-based methods still have a challenge in dealing with non-IID training data of local devices (i.e., learners). In this regard, we study the effects of a variety of hyperparametric conditions under the non-IID environments, to answer important concerns in practical implementations: (i) We first investigate parameter divergence of local updates to explain performance degradation from non-IID data. The origin of the parameter divergence is also found both empirically and theoretically. (ii) We then revisit the effects of optimizers, network depth/width, and regularization techniques; our observations show that the well-known advantages of the hyperparameter optimization strategies could rather yield diminishing returns with non-IID data. (iii) We finally provide the reasons of the failure cases in a categorized way, mainly based on metrics of the parameter divergence.
The paper experimentally studies the reasons for the slow convergence of the Federated Averaging algorithm when the data are non-iid distributed between workers in the multiclass-classification case. Paper performs extensive experimental study and observes that the main reasons for failure are connected to (i) the parameter divergence during the local steps, (ii) steep-fall phenomena when parameters on different nodes are getting close fast, and to the (iii) high training loss. 
My score is weak reject. The paper provides extensive but unclear experimental results. Improving presentation would significantly improve the paper. For example, why in experimental and theoretical study different parameter divergence metrics were used, etc (see below), why different networks use different optimizers. 
Moreover, provided experimental comparison might be unfair. The learning rate is constant throughout all of the experiments, depending only on the optimizer, but not on the neural network architecture. This can affect the final results. 
Concerns and questions that should be addressed:
1. The initial learning rates were not tuned properly. It is set to be the same for different neural network topologies, which might significantly affect the results. What did the choice of initial learning rates is based on? 
2. Why the parameter divergence metric in Definition 1 is not the same as in the theoretical study (Appendix B)? What is the intuition behind Definition 1?
3. Why the divergence of parameters is considered only at the last layer? It seems to hide many important interactions in the other layers. 
4. Some important experimental details --- should be added:
   - At which moment the parameter divergence is computed in the plots? Is it computed at the end of the local iterations right before synchronization? 
   - How the training loss was computed in the plots? before or after synchronization? on the local only or the global data?
   - Which batch size was used? 
   - Improve the figure caption to detail the experimental setup. (e.g. in fig 3. the network architecture was mentioned only for one of the figures, include which optimized was used, etc)
5. In experiments on Fig. 2. and Fig.3 (middle) what is the accuracy for IID baseline? Is the observed phenomena connected to the poor network architecture or to the non-iid data? 
6. In table 5 of the appendix, why experiments use Adam optimizer, but not Momentum SGD as in the main paper to compare the performance of ResNet14 and ResNet20?
7. Better re-prase the definition of the steep fall phenomena, now it is not very clear: in the IID setting parameter divergence values are also sometimes reducing sharply; in the network width study parameters divergence doesn’t experience sudden drop. Also, how does this phenomena (and parameter divergence too) connects to the training loss? 
8. Why for different experiments different baseline models are used? (NetA, NetB, NetC)
Other minor comments: 
- Appendix B, first equation on page 13. (d_q)^t -> (d_q)^t_k; The size of gradient \nabla_w [E ...] is different from the size of (d_q)_k. They cannot be added together.
- page 7, last sentence of the first paragraph: what is the accuracy achieved with Batch Renormalization? Why the reason for accuracy gap is “significant parameter divergence”? on fig. 3 “parameter divergence” is smaller than for the baseline.
- Why the name of the section on page 7 is “excessively high training loss of local updates” if later it is stated that it is actually smaller than for the IID case? 
- Defenition 1, line 4: “the then” -> “the”
- section 3: “A pleasant level of parameter divergence can help to improve generalization” -> where was it shown?
- section 4.2, paragraph 2: what is meant by “hyperparametric methods”?
- section 4.2, paragraph 3: “quantitative increase in a layer level” -> not clear what does it mean.
- page 4, effect of optimizers: what do you refer to as “all model parameters”? 
- page 5, last paragraph: Hinton et al... -> (Hinton et al…). Use \citet(\citep) instead of \cite. 
- why Dropout yields bigger parameter divergence if on Fig 2, right it actually helps? 
- Last line of the page 5. Where was this observed?