CANE: Context-Aware Network Embedding for Relation Modeling
Network embedding (NE) is playing a critical role in network analysis, due to its ability to represent vertices with efficient low-dimensional embedding vectors. However, existing NE models aim to learn a fixed context-free embedding for each vertex, and neglect the diverse roles when interacting with other vertices. In this paper, we assume that one vertex usually shows different aspects when interacting with different neighbor vertices, and should own different embeddings respectively. Therefore, we present ContextAware Network Embedding (CANE), a novel NE model to address this issue. CANE learns context-aware embeddings for vertices with mutual attention mechanism and is expected to model the semantic relationships between vertices more precisely. In experiments, we compare our model with existing NE models on three real-world datasets. Experimental results shows that CANE achieves significant improvement than state-of-the-art methods on link prediction, and comparable performance on vertex classification.
1 Introduction
Network embedding (NE), i.e., network representation learning (NRL), aims to map vertices of a network into a low-dimensional space according to their structural roles in the network. NE provides an efficient and effective way to represent and manage large-scale networks, alleviating the computation and sparsity issues of conventional symbol-based representations. Hence, NE is attracting many research interests in recent years (Perozzi et al., 2014; Tang et al., 2015; Grover and
Leskovec, 2016), and achieves promising performance on many network analysis tasks including link prediction, vertex classification, and community detection.
In real-world social networks, it is intuitive that one vertex may demonstrate various aspects when interacting with different neighbor vertices. For example, a researcher usually collaborates with different partners on diverse research topics (as illustrated in Fig. 1), a social-media user contacts with various friends sharing distinct interests, and a web page links with multiple pages for different purposes. However, most existing NE methods only arrange one single embedding vector to each vertex, and give rise to the following two invertible issues: (1) These methods cannot cope with the aspect transition of a vertex flexibly when interacting with different neighbors. (2) In these models, a vertex tends to force the embeddings of its neighbors close to each other, which may be not case all the time. For example, the left user and right user in Fig. 1, share less common interests, but are learned to be close to each other since they both link to the middle person. This will accordingly
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
make vertex embeddings indiscriminative. To address these issues, we aim to propose a Context-Aware Network Embedding (CANE) framework for modeling relationships between vertices precisely. More specifically, we present CANE on information networks, where each vertex also contains rich external information such as text, labels or other meta-data, and the significance of context is more critical for NE in this scenario. Without loss of generality, we implement CANE on text-based information networks in this paper, which can be easily extended to other types of information networks.
In conventional NE models, each vertex is represented as a static embedding vector, denoted as context-free embedding. On the contrary, CANE assigns dynamic embeddings to a vertex according to different neighbors it interacts with, named as context-aware embedding . Take a vertex u and its neighbor vertex v for example. The contextfree embedding of u remains unchanged when interacting with different neighbors. On the contrary, the context-aware embedding of u is dynamic when confronting different neighbors.
When u interacting with v, their context embeddings with respect to each other are derived from their text information, Su and Sv respectively. For each vertex, we can easily use neural models, such as convolutional neural networks (Blunsom et al., 2014; Johnson and Zhang, 2014; Kim, 2014) and recurrent neural networks (Kiros et al., 2015; Tai et al., 2015), to build context-free text-based embedding. In order to realize context-aware textbased embeddings, we introduce the selective attention scheme and build mutual attention between u and v into these neural models. The mutual attention is expected to guide neural models to emphasize those words that are focused by its neighbor vertices and eventually obtain contextaware embeddings.
Both context-free embeddings and contextaware embeddings of each vertex can be efficiently learned together via concatenation using existing NE methods such as DeepWalk (Perozzi et al., 2014), LINE (Tang et al., 2015) and node2vec (Grover and Leskovec, 2016).
We conduct experiments on three real-world datasets of different areas. Experimental results on link prediction reveal the effectiveness of our framework as compared to other state-of-the-art methods. The results suggest that, context-aware
embeddings are critical for network analysis, especially for those tasks concerning about complicated interactions between vertices such as link prediction. We also explore the performance of our framework via vertex classification and case studies, which again confirms the flexibility and superiority of our models.
2 Related Work
With the rapid growth of large-scale social networks, network embedding, i.e. network representation learning has been proposed as a critical technique for network analysis tasks.
In recent years, there have been a large number of NE models proposed to learn efficient vertex embeddings (Tang and Liu, 2009; Cao et al., 2015; Wang et al., 2016). For example, DeepWalk (Perozzi et al., 2014) performs random walks over networks and introduces an efficient word representation learning model, Skip-Gram (Mikolov et al., 2013a), to learn network embeddings. LINE (Tang et al., 2015) optimizes the joint and conditional probabilities of edges in large-scale networks to learn vertex representations. Node2vec (Grover and Leskovec, 2016) modifies the random walk strategy in DeepWalk into biased random walks to explore the network structure more efficiently. Nevertheless, most of these NE models only encode the structural information into vertex embeddings, without considering heterogenous information accompanied with vertices in real-world social networks.
To address this issue, researchers make great efforts to incorporate heterogenous information into conventional NE models. For instance, Yang et al. (2015) present text-associated DeepWalk (TADW) to improve matrix factorization based DeepWalk with text information. Tu et al. (2016) propose max-margin DeepWalk (MMDW) to learn discriminative network representations by utilizing labeling information of vertices. Chen et al. (2016) propose group-enhanced network embedding (GENE) to integrate existing group information in NE. Sun et al. (2016) regard text content as a special kind of vertices, and propose contextenhanced network embedding (CENE) through leveraging both structural and textural information to learn network embeddings.
To the best of our knowledge, all existing NE models focus on learning context-free embeddings, but ignore the diverse roles when a vertex
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
interacts with others. In contrast, we assume that a vertex has different embeddings according to which vertex it interacts with, and propose CANE to learn context-aware vertex embeddings.
3 Problem Formulation
We first give basic notations and definitions in this work. Suppose there is an information network G = (V,E, T ), where V is the set of vertices, E ⊆ V ×V are edges between vertices, and T denotes the text information of vertices. Each edge eu,v ∈ E represents the relationship between two vertices (u, v), with an associated weight wu,v. Here, the text information of a specific vertex v ∈ V is represented as a word sequence Sv = (w1, w2, . . . , wnv), where nv = |Sv|. NRL aims to learn a low-dimensional embedding v ∈ Rd for each vertex v ∈ V according to its network structure and associated information, e.g. text and labels. Note that, d |V | is the dimension of representation space.
Definition 1. Context-free Embeddings: Conventional NRL models learn context-free embedding for each vertex. It means the embedding of a vertex is fixed and won’t change with respect to its context information (i.e., another vertex it interacts with).
Definition 2. Context-aware Embeddings: Different from existing NRL models that learn context-free embeddings, CANE learns various embeddings for a vertex according to its different contexts. Specifically, for an edge eu,v, CANE learns context-aware embeddings v(u) and u(v).
4 The Method
4.1 Overall Framework
To take full use of both network structure and associated text information, we propose two types of embeddings for a vertex v, i.e., structurebased embedding vs and text-based embedding vt. Structure-based embedding is able to capture the information in the network structure, while text-based embedding can capture the textual meanings lying in the associated text information. With these embeddings, we can simply concatenate them and obtain the vertex embeddings as v = vs⊕vt, where ⊕ indicates the concatenation operation. Note that, the text-based embedding vt can be either context-free or context-aware, which will be introduced detailedly in section 4.4 and 4.5
respectively. When vt is context-aware, the overall vertex embeddings v will be context-aware as well.
With above definitions, CANE aims to maximize the overall objective of edges as follows:
L = ∑ e∈E L(e). (1)
Here, the objective of each edge L(e) consists of two parts as follows:
L(e) = Ls(e) + Lt(e), (2)
where Ls(e) denotes the structure-based objective and Lt(e) represents the text-based objective.
In the following part, we give detailed introduction to the two objectives respectively.
4.2 Structure-based Objective
Without loss of generality, we assume the network is directed, as an undirected edge can be considered as two directed edges with opposite directions and equal weights.
Thus, the structure-based objective aims to measure the log-likelihood of a directed edge using the structure-based embeddings as
Ls(e) = wu,v log p(v s|us). (3)
Following LINE (Tang et al., 2015), we define the conditional probability of v generated by u in Eq. (3) as
p(vs|us) = exp(u s · vs)∑
z∈V exp(u s · zs)
. (4)
4.3 Text-based Objective
Vertices in real-world social networks usually accompany with associated text information. Therefore, we propose the text-based objective to take advantage of these text information, as well as learn text-based embeddings for vertices.
The text-based objective Lt(e) can be defined with various measurements. To be compatible with Ls(e), we define Lt(e) as follows:
Lt(e) = α ·Ltt(e) + β ·Lts(e) + γ ·Lst(e), (5)
where α, β and γ control the weights of various parts, and
Ltt(e) = wu,v log p(v t|ut), Lts(e) = wu,v log p(v t|us), Lst(e) = wu,v log p(v s|ut).
(6)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
The conditional probabilities in Eq. (6) map the two types of vertex embeddings into the same representation space, but do not enforce them to be identical for the consideration of their own characteristics. Similarly, we employ softmax function for calculating the probabilities, as in Eq. (4).
The structure-based embeddings are regarded as parameters, the same as in conventional NE models. But for text-based embeddings, we intend to obtain them from associated text information of vertices. Besides, the text-based embeddings can be obtained either in context-free ways or in context-aware ones. In the following sections, we will give detailed introduction respectively.
4.4 Context-Free Text Embedding
There has been a variety of neural models to obtain text embeddings from a word sequence, such as convolutional neural networks (CNN) (Blunsom et al., 2014; Johnson and Zhang, 2014; Kim, 2014) and recurrent neural networks (RNN) (Kiros et al., 2015; Tai et al., 2015).
In this work, we investigate different neural networks for text modeling, including CNN, Bidirectional RNN (Schuster and Paliwal, 1997) and GRU (Cho et al., 2014), and employ the best performed CNN, which can capture the local semantic dependency among words.
Taking the word sequence of a vertex as input, CNN obtains the text-based embedding through three layers, i.e. looking-up, convolution and pooling.
Looking-up. Given a word sequence S = (w1, w2, . . . , wn), the looking-up layer transforms each word wi ∈ S into its corresponding word embedding wi ∈ Rd ′ and obtains embedding sequence as S = (w1,w2, . . . ,wn). Here, d′ indicates the dimension of word embeddings.
Convolution. After looking-up, the convolution layer extracts local features of input embedding sequence S. To be specific, it performs convolution operation over a sliding window of length l using a convolution matrix C ∈ Rd×(l×d′) as follows:
xi = C · Si:i+l−1 + b, (7)
where Si:i+l−1 denotes the concatenation of word embeddings within the i-th window and b is the bias vector. Note that, we add zero padding vectors (Hu et al., 2014) at the edge of the sentence.
Max-pooling. To obtain the text embedding vt, we operate max-pooling and non-linear transfor-
mation over {xi0, . . . ,xin} as follows:
ri = tanh(max(x i 0, . . . ,x i n)), (8)
At last, we encode the text information of a vertex with CNN and obtain its text-based embedding vt = [r1, . . . , rd]
T . As vt is irrelevant to the other vertices it interacts with, we name it as contextfree text embedding.
4.5 Context-Aware Text Embedding
As stated before, we assume that a specific vertex plays different roles when interacting with others vertices. In other words, each vertex should have its own points of focus about a specific vertex, which leads to its context-aware text embeddings.
To achieve this, we employ mutual attention to obtain context-aware text embedding. It enables the pooling layer in CNN to be aware of the vertex pair in an edge, in a way that text information from a vertex can directly affect the text embedding of the other vertex, and vice versa.
In Fig. 2, we give an illustration of the generating process of context-aware text embedding. Given an edge eu,v with two corresponding text sequences Su and Sv, we can get the matrices P ∈ Rd×m and Q ∈ Rd×n through convolution layer. Here, m and n represent the lengths of Su
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
and Sv respectively. By introducing an attentive matrix A ∈ Rd×d, we compute the correlation matrix F ∈ Rm×n as follows:
F = tanh(PTAQ). (9)
Note that, each element Fi,j in F represents the pair-wise correlation score between two hidden vectors, i.e., Pi and Qj .
After that, we conduct pooling operations along rows and columns of F to generate the importance vectors, named as row-pooling and column pooling respectively. According to our experiments, mean-pooling performs better than max-pooling. Thus, we employ mean-pooling operation as follows:
gpi = mean(Fi,1, . . . ,Fi,n), gqi = mean(F1,i, . . . ,Fm,i). (10)
The importance vectors of P and Q are obtained as gp = [gp1 , . . . , g p m]T and gq = [gq1, . . . , g q n]T .
Next, we employ softmax function to transform importance vectors gp and gq to attention vectors ap and aq. For instance, the i-th element of ap is formalized as follows:
api = exp(gpi )∑
j∈[1,m] exp(g p j ) . (11)
At last, the context-aware text embeddings of u and v are computed as
ut(v) = Pa p, vt(u) = Qa q.
(12)
Now, given an edge (u, v), we can obtain the context-aware embeddings of vertices with their structure embeddings and context-aware text embeddings as u(v) = us⊕ut(v) and v(u) = v s⊕vt(u).
4.6 Optimization of CANE
According to Eq. (3) and Eq. (6), CANE aims to maximize several conditional probabilities between u ∈ {us,ut(v)} and v ∈ {v
s,vt(u)}. It is intuitive that optimizing the conditional probability using softmax function is computationally expensive. Thus, we employ negative sampling (Mikolov et al., 2013b) and transform the objective into the following form:
log σ(uT ·v)+ k∑
i=1
Ez∼P (v)[log σ(−uT ·z)], (13)
where k is the number of negative samples and σ represents the sigmoid function. P (v) ∝ dv3/4 denotes the distribution of vertices, where dv is the out-degree of v.
Afterwards, we employ Adam (Kingma and Ba, 2015) to optimize the transformed objective.
5 Experiments
In order to investigate the effectiveness of CANE on modeling relationships between vertices, we conduct experiments of link prediction on several real-world datasets. Besides, we also employ vertex classification to verify whether contextaware embeddings of a vertex can compose a highquality context-free embedding in return.
5.1 Datasets
Datasets Cora HepTh Zhihu
#Vertices 2, 277 1, 038 10, 000 #Edges 5, 214 1, 990 43, 894 #Labels 7 − −
Table 1: Statistics of Datasets.
We select three real-world network datasets as follows:
Cora1 is a typical paper citation network constructed by (McCallum et al., 2000). After filtering out papers without text information, there are 2, 277 machine learning papers in this network, which are divided into 7 categories.
HepTh2 (High Energy Physics Theory) is another citation network from arXiv3 released by (Leskovec et al., 2005). We filter out papers without abstract information and retain 1, 038 papers at last.
Zhihu4 is the largest online Q&A website in China. Users follow each other and answer questions in this site. We randomly crawl 10, 000 active users from Zhihu, and take the descriptions of their concerned topics as text information.
The detailed statistics are listed in Table 1.
5.2 Baselines
We employ the following methods as baselines:
Structure-only: DeepWalk (Perozzi et al., 2014) performs random walks over networks and employ Skip-Gram 1https://people.cs.umass.edu/∼mccallum/data.html 2https://snap.stanford.edu/data/cit-HepTh.html 3https://arxiv.org/ 4https://www.zhihu.com/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
%Removed edges 15% 25% 35% 45% 55% 65% 75% 85% 95%
DeepWalk 56.0 63.0 70.2 75.5 80.1 85.2 85.3 87.8 90.3 LINE 55.0 58.6 66.4 73.0 77.6 82.8 85.6 88.4 89.3 node2vec 55.9 62.4 66.1 75.0 78.7 81.6 85.9 87.3 88.2
Naive Combination 72.7 82.0 84.9 87.0 88.7 91.9 92.4 93.9 94.0 TADW 86.6 88.2 90.2 90.8 90.0 93.0 91.0 93.4 92.7 CENE 72.1 86.5 84.6 88.1 89.4 89.2 93.9 95.0 95.9
CANE (text only) 78.0 80.5 83.9 86.3 89.3 91.4 91.8 91.4 93.3 CANE (w/o attention) 85.8 90.5 91.6 93.2 93.9 94.6 95.4 95.1 95.5
CANE 86.8 91.5 92.2 93.9 94.6 94.9 95.6 96.6 97.7
Table 2: AUC values on Cora. (α = 1.0, β = 0.3, γ = 0.3)
model (Mikolov et al., 2013a) to learn vertex embeddings.
LINE (Tang et al., 2015) learns vertex embeddings in large-scale networks using first-order and second-order proximities.
Node2vec (Grover and Leskovec, 2016) proposes a biased random walk algorithm based on DeepWalk to explore neighborhood architecture more efficiently.
Structure and Text: Naive Combination: We simply concatenate the best-performed structure-based embeddings with CNN based embeddings to represent the vertices.
TADW (Yang et al., 2015) employs matrix factorization to incorporate text features of vertices into network embeddings.
CENE (Sun et al., 2016) leverages both structure and textural information by regarding text content as a special kind of vertices, and optimizes the probabilities of heterogenous links.
5.3 Evaluation Metrics and Experiment Settings
For link prediction, we adopt a standard evaluation metric AUC (Hanley and McNeil, 1982), which represents the probability that vertices in a random unobserved link are more similar than those in a random nonexistent link.
For vertex classification, we employ L2regularized logistic regression (L2R-LR) (Fan et al., 2008) to train classifiers, and evaluate the classification accuracies of various methods.
To be fair, we set the embedding dimension to 200 for all methods. In LINE, we set the number of negative samples to 5; we learn the 100 dimensional first-order and second-order embeddings respectively, and concatenate them to form the 200 dimensional embeddings. In node2vec, we employ grid search and select the best per-
formed hyper-parameters for training. We also employ grid search to set the hyper-parameters α, β and γ in CANE. Besides, we set the number of negative samples k to 1 in CANE to speed up the training process. To demonstrate the effectiveness of considering attention mechanism and two types of objectives in Eqs. (3) and (6), we design three versions of CANE for evaluation, i.e., CANE with text only, CANE without attention and CANE.
5.4 Link Prediction
As shown in Table 2, Table 3 and Table 4, we evaluate the AUC values while removing different ratios of edges on Cora, HepTh and Zhihu respectively. Note that, when we only keep 5% edges for training, most vertices are isolated, which results in the poor and meaningless performance of all the methods. Thus, we omit the results under this training ratio. From these tables, we have the following observations:
(1) Our proposed CANE consistently achieves significant improvement comparing to all the baselines on all different datasets and different training ratios. It indicates the effectiveness of CANE when applied to link prediction task, and verifies that CANE has the capability of modeling relationships between vertices precisely.
(2) What calls for special attention is that, both CENE and TADW exhibit unstable performance under various training ratios. Specifically, CENE performs poorly under small training ratios, because it reserves much more parameters (e.g., convolution kernels and word embeddings) than TADW, which need more data for training. Different from CENE, TADW performs much better under small training ratios, because DeepWalk based methods can explore the sparse network structure well through random walks even with limited edges. However, it achieves poor performance
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
695
696
697
698
699
%Removed edges 15% 25% 35% 45% 55% 65% 75% 85% 95%
DeepWalk 55.2 66.0 70.0 75.7 81.3 83.3 87.6 88.9 88.0 LINE 53.7 60.4 66.5 73.9 78.5 83.8 87.5 87.7 87.6 node2vec 57.1 63.6 69.9 76.2 84.3 87.3 88.4 89.2 89.2
Naive Combination 78.7 82.1 84.7 88.7 88.7 91.8 92.1 92.0 92.7 TADW 87.0 89.5 91.8 90.8 91.1 92.6 93.5 91.9 91.7 CENE 86.2 84.6 89.8 91.2 92.3 91.8 93.2 92.9 93.2
CANE (text only) 83.8 85.2 87.3 88.9 91.1 91.2 91.8 93.1 93.5 CANE (w/o attention) 84.5 89.3 89.2 91.6 91.1 91.8 92.3 92.5 93.6
CANE 90.0 91.2 92.0 93.0 94.2 94.6 95.4 95.7 96.3
Table 3: AUC values on HepTh. (α = 0.7, β = 0.2, γ = 0.2)
%Removed edges 15% 25% 35% 45% 55% 65% 75% 85% 95%
DeepWalk 56.6 58.1 60.1 60.0 61.8 61.9 63.3 63.7 67.8 LINE 52.3 55.9 59.9 60.9 64.3 66.0 67.7 69.3 71.1 node2vec 54.2 57.1 57.3 58.3 58.7 62.5 66.2 67.6 68.5
Naive Combination 55.1 56.7 58.9 62.6 64.4 68.7 68.9 69.0 71.5 TADW 52.3 54.2 55.6 57.3 60.8 62.4 65.2 63.8 69.0 CENE 56.2 57.4 60.3 63.0 66.3 66.0 70.2 69.8 73.8
CANE (text only) 55.6 56.9 57.3 61.6 63.6 67.0 68.5 70.4 73.5 CANE (w/o attention) 56.7 59.1 60.9 64.0 66.1 68.9 69.8 71.0 74.3
CANE 56.8 59.3 62.9 64.5 68.9 70.4 71.4 73.6 75.4
Table 4: AUC values on Zhihu. (α = 1.0, β = 0.3, γ = 0.3)
under large ones, as its simplicity and the limitation of bag-of-words assumption. On the contrary, CANE has a stable performance on various situations. It demonstrates the flexibility and robustness of CANE.
(3) By introducing attention mechanism, the learnt context-aware embeddings obtain considerable improvements than the ones without attention. It verifies our assumption that a specific vertex should play different roles when interacting with other vertices, and thus benefits the relevant link prediction task.
To summarize, all the above observations demonstrate that CANE is able to learn highquality context-aware embeddings, which are conducive to estimating the relationship between vertices precisely. Moreover, the experimental results on link prediction task state the effectiveness and robustness of CANE.
5.5 Vertex Classification
In CANE, we obtain various embeddings of a vertex according to the vertex it connects to. It’s intuitive that the obtained context-aware embeddings are naturally applicable to link prediction task. However, network analysis tasks, such as vertex classification and clustering, require a global embedding, rather than several context-aware embed-
dings for each vertex. To demonstrate the capability of CANE to solve these issues, we generate the global embedding of a vertex u by simply averaging all the contextaware embeddings as follows:
u = 1
N ∑ (u,v)|(v,u)∈E u(v),
where N indicates the number of context-aware embeddings of u.
50
55 60 65 70 75 80 85 90 95
100
De ep
W alk LI NE
no de
2v ec NC TA DW CE NE
CA NE
(te xt
on ly)
CA NE
(w /0
at ten
tio n) CA NE
A cc
ur ac
y (×
1 00
)
With the generated global embeddings, we conduct experiments of vertex classification on Cora. As shown in Fig. 3, we observe that:
(1) CANE achieves comparable performance with state-of-the-art model CENE. It states that the
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
learnt context-aware embeddings can transform into high-quality context-free embeddings through simple average operation, which can be further employed to other network analysis tasks.
(2) With the introduction of mutual attention mechanism, CANE has an encouraging improvement than the one without attention, which is in accordance with the results of link prediction. It denotes that CANE is flexible to various network analysis tasks.
5.6 Case Study
To demonstrate the significance of mutual attention on selecting meaningful features from text information, we visualize the heat maps of two vertex pairs in Fig. 4. Note that, every word in this figure accompanies with various background colors. The stronger the background color is, the larger the weight of this word is. The weight of each word is calculated according to the attention weights as follows.
For each vertex pair, we can get the attention weight of each convolution window according to Eq. (11). To obtain the weights of words, we assign the attention weight to each word in this window, and add the attention weights of a word together as its final weight.
We select three connected vertices in Cora for example, denoted as A, B and C. From Fig. 4, we observe that, though there exists citation relations with identical paper A, paper B and C concern about different parts of A. The attention weights over A in edge #1 are assigned to “reinforcement learning”. On the contrary, the weights in edge #2 are assigned to “machine learning’”, “supervised learning algorithms” and “complex stochastic models”. Moreover, all these key elements in A can find corresponding words in B and C. It’s intuitive that these key elements give an exact explanation on the citation relations. The discovered significant correlations between vertex pairs reflects the effectiveness of mutual attention mechanism, as well as the capability of CANE for modeling relations precisely.
6 Conclusion and Future Work
In this paper, we propose the concept of ContextAware Network Embedding (CANE) for the first time, which aims to learn various context-aware embeddings for a vertex according to the neighbors it interacts with. Specifically, we implement
CANE on text-based information networks with proposed mutual attention mechanism, and conduct experiments on several real-world information networks. Experimental results on link prediction demonstrate that CANE is effective for modeling the relationship between vertices. Besides, the learnt context-aware embeddings can compose high-quality context-free embeddings.
We will explore the following directions in future:
(1) We have investigated the effectiveness of CANE on text-based information networks. In future, we will strive to implement CANE on wider variety of information networks with multi-modal data, such as labels, images and so on.
(2) CANE encodes latent relations between vertices into their context-aware embeddings. Furthermore, there usually exist explicit relations in social networks (e.g., families, friends and colleagues relations between social network users), which are expected to be critical to NE. Thus, we want to explore how to incorporate and predict these explicit relations between vertices in NE.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
This paper addresses the network embedding problem by introducing a neural network model which uses both the network structure and associated text on the nodes, with an attention model to vary the textual representation based on the text of the neighboring nodes.  - Strengths:  The model leverages both the network and the text to construct the latent representations, and the mutual attention approach seems sensible.  A relatively thorough evaluation is provided, with multiple datasets, baselines, and evaluation tasks.  - Weaknesses:  Like many other papers in the  network embedding  literature, which use neural network techniques inspired by word embeddings to construct latent representations of nodes in a network, the previous line of work on statistical probabilistic modeling of networks is ignored.  In particular, all  network embedding  papers need to start citing, and comparing to, the work on the latent space model of Peter Hoff et al., and subsequent papers in both statistical and probabilistic machine learning publication venues:  P.D. Hoff, A.E. Raftery, and M.S. Handcock. Latent space approaches to social network analysis. J. Amer. Statist. Assoc., 97(460):1090 1098, 2002.  This latent space network model, which embeds each node into a low-dimensional latent space, was written as far back as 2002, and so it far pre-dates neural network-based network embeddings.  Given that the aim of this paper is to model differing representations of social network actors' different roles, it should really cite and compare to the mixed membership stochastic blockmodel (MMSB):  Airoldi, E. M., Blei, D. M., Fienberg, S. E., & Xing, E. P. (2008). Mixed membership stochastic blockmodels. Journal of Machine Learning Research.  The MMSB allows each node to randomly select a different  role  when deciding whether to form each edge.  - General Discussion:  The aforementioned statistical models do not leverage text, and they do not use scalable neural network implementations based on negative sampling, but they are based on well-principled generative models instead of heuristic neural network objective functions and algorithms.  There are more recent extensions of these models and inference algorithms which are more scalable, and which do leverage text.  Is the difference in performance between CENE and CANE in Figure 3 statistically insignificant? (A related question: were the experiments repeated more than once with random train test splits?)  Were the grid searches for hyperparameter values, mentioned in Section 5.3, performed with evaluation on the test set (which would be problematic), or on a validation set, or on the training set?