In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs. We prove that DrGCNs have the effect of stabilizing the training process by connecting our problem to the theory of the mean field. However, practically, we find that the degrees DrGCNs help vary severely on different datasets. We revisit the problem and develop a new measure K to quantify the effect. This measure guides when we should use dimensional reweighting in GCNs and how much it can help. Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs. The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants. Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches. Significant improvements can also be observed on a large scale industrial dataset.
This paper proposes a method, known as DrGCN, for reweighting the different dimensions of the node representations in graph convolutional networks (GCN). Specifically, the representation of every node is element-wise multiplied with a weight vector, which is parameterized as a function of the average input node representation, where the function is a two-layer neural network. 
At a conceptual level, this is similar to various existing normalization schemes, such as batch normalization and weight normalization. While it is claimed in Section 4.3 that the difference is that “[batch normalization] reduces variance between samples, while DrGCN reduces variance between dimensions”, I am not sure if this characterization is accurate. Batch normalization actually makes each dimension have unit (sample) variance and so does not make the variance of each dimension small. What it does is to make the sample variance of each dimension equal, which is also what DrGCN tries to do, i.e.: reduce variance across dimensions (since samples in the case of GCNs are the representations of different nodes). DrGCN is also similar to weight normalization because like weight normalization, DrGCN learns a transformation on top of the vanilla representation (in the case of weight normalization, the vanilla representation is the normalized weight vector; in the case of DrGCN, the vanilla representation is the node representation before reweighting). The conceptual contribution therefore seems incremental. 
Incremental conceptual contributions would be fine if (1) they result in a surprising theoretical result, or if (2) they result in a surprising improvement in empirical performance. Unfortunately, neither seems to be demonstrated in this paper. 
In the theoretical analysis, there are various occurrences of unjustified leaps of logic; as a result, what is claimed to be shown by the analysis is different from what is actually shown, and it is unclear what is actually shown is substantially related to the proposed method. For example, in Section 3.1, the paper says that “GCNs are different from fully-connected networks only in \tilde{A}, and degrade to fully-connected networks when \tilde{A} = I. So, our analysis can be somehow be generalized to DrGCNs.” The first sentence is true; in other words, it says that fully-connected networks are a special case of GCNs when \tilde{A} = I. However, the second sentence does not follow - just showing a special case (which is what the subsequent analysis does) does not say much about the general case. Relatedly, the architecture that is analyzed is of the form H^l = W^l \phi(S^l H^{l-1}) + b^l for l = 1, …, k, whereas the architecture that is proposed is H^l = \phi(W^l S^l H^{l-1} \tilde{A}^{l} + b^{l}) for l = 1, …, k. The latter cannot be cast as the former unless \tilde{A} is diagonal. Also, the caveat of the mean field approximation are not stated - whatever result that is shown is only valid at the infinite width limit, which is different from what is claimed in the abstract, which says that "We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.” Additionally, the analysis is done in the case where S is directly parameterized, whereas the proposed method parameterizes S as the output of a two-layer neural network. I presume the reason why latter was done in practice because it worked better empirically. Because this is not explained by the analysis, the theory is incomplete, and so this caveat should be clearly stated in the abstract. 
In the experimental results, the performance improvement of DrGCN over layer normalization is not statistically significant. Also, DrGCN is only compared to other normalization schemes (batch normalization and layer normalization) on one dataset, and so there is insufficient evidence to conclude that DrGCN generally works better than existing methods empirically, 
Additionally, in section 2.1, it is claimed that “sampling-based GCNs still lie within the framework of equation (2), as we can set all unsampled edges to 0 in \tilde{A}”. This does not seem to be accurate, because in sample-based GCNs, different edges are sampled in every minibatch, and so there is no fixed choice of \tilde{A} that makes these GCNs equivalent to the formulation in eq. (2). 
Also, if the goal is to reduce variance across dimensions (as the paper claims in Section 4.3), why was the average node representation fed into the two-layer neural network rather than its reciprocal?