Network representation learning (NRL) is a powerful technique for learning lowdimensional vector representation of high-dimensional and sparse graphs. Most studies explore the structure and metadata associated with the graph using random walks and employ an unsupervised or semi-supervised learning schemes. Learning in these methods is context-free, because only a single representation per node is learned. Recently studies have argued on the sufficiency of a single representation and proposed a context-sensitive approach that proved to be highly effective in applications such as link prediction and ranking. However, most of these methods rely on additional textual features that require RNNs or CNNs to capture high-level features or rely on a community detection algorithm to identify multiple contexts of a node. In this study, without requiring additional features nor a community detection algorithm, we propose a novel context-sensitive algorithm called GAP that learns to attend on different parts of a node's neighborhood using attentive pooling networks. We show the efficacy of GAP using three real-world datasets on link prediction and node clustering tasks and compare it against 10 popular and state-ofthe-art (SOTA) baselines. GAP consistently outperforms them and achieves up to ≈ 9% and ≈ 20% gain over the best performing methods on link prediction and clustering tasks, respectively.
Summary:
The proposed paper adapts Attentive Pooling Network (ATN) for graphs, by noting that the order of
neighbors of any node in a graph does not matter unlike the neighboring words in a sentence (for
which APN was developed). A simple modification of removing the layer which encodes higher order
sequential properties in APN, eg n-gram like statistics, the APN is adapted to GAP. This allows
encoding context from the neighboring nodes for a specific pair to be compared. Applications on Link
Prediction and Node Clustering are demonstrated on three benchmark datasets.
Detailed comments:
- The paper is simple and easy to read and the modification of APN to GAP is also small while being
well motivated and empirically important.
- There are some confusions in the writing: p4 first paragraph says that "In principle one can learn
using all pairs of nodes, however that is not scalable, and hence we restrict learning between pairs
in E." However from and around Eq2 it appears that r_s.r_t^- is also computed, where (s,r^-) is not
an edge.
- Also the statement below Eq2 "The goal is to learn, in an unsupervised fashion..." is not correct
  specially for link prediction task, as having a training graph (with edges) is having annotations.
        
- Although, the removal of the `encoding' step from APN makes sense, it would also be possible that
  the encoding operation is learned to be an identity function automatically if the training data is
  presented appropriately. I would assume that learning in such a case might take longer, and might
  not achieve the performance achieved by GAP. An ablation experiment could be reported to support
  GAP further where encoder is kept in APN, but at the training same pair is presented multiple
  times with the order of the neighborhoods randomized.      
       
- Similar to the experiment above, the order of the nodes could be arbitrarily fixed by some simple
  logic. Eg order the neighboring nodes sorted by the feature similarity with the current node. This
  can then be fed to APN and a baseline could be reported.
- I do not directly work with Link Prediction or Node Clustering tasks, so am not familiar with the
  literature and the baselines reported. However, it might be that many baselines do not use a
  discriminative objective. Perhaps just using the discriminative objective gives a lot of boost and
  using the context from neighbors is not very important? From Fig2 and Fig3a it seems that
  neighborhood size does not make a big difference? This should also be ablated. One experiment
  could be to use just the current feature and have a small MLP on it, and learn using the proposed
  discriminative objective. If any other experiment can be designed in similar lines, it should be
  included as well.
- Why are run time comparison given on simulated data and not real data?
- The experimental setup for training edge selection should be detailed more. In particular, I
  would recommend that true generalization in terms of nodes and edges should be ensured, by having
  no test node (i) appear in the train set and (ii) has an edge which connects it to a training
  node. If this is not used then the alternative scheme used should be explained and argued for.
I am not a direct expert in the area and felt that the paper was somewhat lacking. I am putting my initial rating
to the conservative side. I am very open to revising the rating based on the author responses and the other 
reviewers' comments.