Learning latent representations of nodes in graphs is an important and ubiquitous task with widespread applications such as link prediction, node classification, and graph visualization. Previous methods on graph representation learning mainly focus on static graphs, however, many real-world graphs are dynamic and evolve over time. In this paper, we present Dynamic Self-Attention Network (DySAT), a novel neural architecture that operates on dynamic graphs and learns node representations that capture both structural properties and temporal evolutionary patterns. Specifically, DySAT computes node representations by jointly employing self-attention layers along two dimensions: structural neighborhood and temporal dynamics. We conduct link prediction experiments on two classes of graphs: communication networks and bipartite rating networks. Our experimental results show that DySAT has a significant performance gain over several different stateof-the-art graph embedding baselines.
This is a well-written paper studying the important problem of dynamic network embedding. Please find below some pros and cons of this paper.
Pros:
* Studies the important problem of network embedding under a more realistic setting (i.e., nodes & edges evolve over time).
* Introduces an interesting architecture that uses two forms of attention: structural and temporal.
* Demonstrated the effectiveness of the temporal layers through additional experiments (in appendix) and also introduced a variant of their proposed approach which can be trained incrementally using only the last snapshot.
Cons:
* The authors compared against several dynamic & static graph embedding approaches. If we disregard the proposed approach (DySAT), the static methods seem to match and even, in some cases, beat the dynamic approaches on the compared temporal graph datasets. The authors should compare against stronger baselines for static node embedding, particularly GAT which introduced the structural attention that DySAT uses to show that the modeling of temporal dependencies is necessary/useful. Please see [1] for an easy way to train GCN/GAT for link prediction.
* There are actually quite a number of work done on network embedding on dynamic graphs including [2-4]. In particular, [2-3] support node attributes as well as the addition/deletion of nodes & edges. The author should also compare against these work.
* The concept of temporal attention is quite interesting. However, the authors do not provide more analysis on this. For one, I am interested to see how the temporal attention weights are distributed. Are they focused on the more recent snapshots? If so, can we simply retain the more relevant recent information and train a static network embedding approach? Or are the attention weights distributed differently?
[1] Modeling Polypharmacy Side Effects with Graph Convolutional Networks. Zitnik et. al. BioInformatics 2018. 
[2] Attributed Network Embedding for Learning in a Dynamic Environment. Li et. al. In Proc. CIKM '17. 
[3] Streaming Link Prediction on Dynamic Attributed Networks. Li et. al. In Proc. WSDM '18. 
[4] Continuous-Time Dynamic Network Embeddings. Nguyen et. al. In Comp. Proc. WWW '18.