Diffusion Maps for Textual Network Embedding
Textual network embedding leverages rich text information associated with the network to learn low-dimensional vectorial representations of vertices. Rather than using typical natural language processing (NLP) approaches, recent research exploits the relationship of texts on the same edge to graphically embed text. However, these models neglect to measure the complete level of connectivity between any two texts in the graph. We present diffusion maps for textual network embedding (DMTE), integrating global structural information of the graph to capture the semantic relatedness between texts, with a diffusion-convolution operation applied on the text inputs. In addition, a new objective function is designed to efficiently preserve the high-order proximity using the graph diffusion. Experimental results show that the proposed approach outperforms state-of-the-art methods on the vertex-classification and link-prediction tasks.
The main idea of this paper is to use the diffusion convolutional operator to learn text embedding that takes into account the global influence of the whole graph. It then incorporates the diffusion process in the loss function to capture high-order proximity. In contrast, previous works either neglect the semantic distance indicated from the graph, or fails to take into account the similarities of context influenced by global structural information. The author then conducts experiments on the task of multi-label classification of text and link prediction and shows that the proposed model outperforms the baselines.   Strength: The high level idea of of this paper is good, and the method is novel. The diffusion convolutional operator is a good idea to softly incorporate the global factor of the whole graph.   Weakness:  The major quality problem of this paper is clarity. In terms of clarity, there are several confusing places in the paper, especially in equation 9, 10, 11, 12.   1) What is s_{i,j} in these equations? In definition 1, the author mentions that s_{i,j} denotes edge weights in the graph, but what are their values exactly in the experiments?  Are they 0/1 or continuous values?   2) How is the diffusion map computed for structural embedding in 10 and 12? Is it using equation 1 only with the learned structural embedding and without text embedding?  3) Why is the diffusion convolution operator only applied to text embedding? Can it also be applied to structural embedding?  On the other hand, if the author wants to capture global information in the graph as claimed between line 191 and line 194, why not directly use the diffusion map in equation (1) on text embedding instead of applying the diffusion convolution operator in 4.2? It's confusing to me the relationship between equation (1) and equation (5), (6) and (7) in section 4.2. In other words, there are two methods that could capture the global information: equation (1), and equation (5)(6)(7). Equation (1) is applied to structural embedding in equation (10) and (12); equation (5)(6)(7) are applied to textual embeddings. The author should explain why they do so.   4) I wonder whether the structural embedding is really necessary in this case, since in the learning process, the structural embedding just involves a embedding table lookup. The author does not explain the point of using a structural embedding, especially in such a way. What if just use diffusion text embedding? I don't see any experimental results proving the effectiveness of structural embedding in Table 1.   5) What's the motivation of each part in equation (8)? For example, what's the motivation of maximizing the probability of textual embedding of vertex i conditioned on the diffusion map of structural embedding of vertex j in equation (12)?   6) In line 135, the author says "Initially the network only has a few active vertices, due to sparsity." How is "active vertices" defined here?   7) In 5.2, when the author trains the SVM classifier, do they also fine-tune the embeddings or just freeze them? There are many existing CNN and RNN based neural classifier for text classification. What if just use any of those off-the-shelf methods on the text embedding without the diffusion process and fine-tune those embedding? This is typically a strong baseline for text classification, but there is no corresponding experimental results in Figure 5.   8) In line 234, how those objective function weights obtained? Are they tuned on any development set? Has the author tried only using a subset of those objectives? It's not clear how important each of those four objectives is.   9) In line 315, the author attributes the result of Table 3 to "both structure and text information". However, the fact that the method picks vertex 3 is due to diffusion convolution operator, as explained in line 313.  Does the "structure" here mean the diffusion convolution operator or the structural embedding?