On Graph Reconstruction via Empirical Risk Minimization: Fast Learning Rates and Scalability
The problem of predicting connections between a set of data points finds many applications, in systems biology and social network analysis among others. This paper focuses on the graph reconstruction problem, where the prediction rule is obtained by minimizing the average error over all n(n − 1)/2 possible pairs of the n nodes of a training graph. Our first contribution is to derive learning rates of order O P (log n/n) for this problem, significantly improving upon the slow rates of order O P (1/ √ n) established in the seminal work of Biau and Bleakley (2006). Strikingly, these fast rates are universal, in contrast to similar results known for other statistical learning problems (e.g., classification, density level set estimation, ranking, clustering) which require strong assumptions on the distribution of the data. Motivated by applications to large graphs, our second contribution deals with the computational complexity of graph reconstruction. Specifically, we investigate to which extent the learning rates can be preserved when replacing the empirical reconstruction risk by a computationally cheaper Monte-Carlo version, obtained by sampling with replacement B n 2 pairs of nodes. Finally, we illustrate our theoretical results by numerical experiments on synthetic and real graphs.
This paper considers a generative model of a random graph. Graphs are generated as follows. We start with a set of vertices. For each vertex, we independently and identically sample a feature from a vector space according to a law \mu. A stochastic kernel, \eta, determines the Bernoulli probability of an edge being present, conditioned on two (unordered) vertex features. For each pair of vertices, we use \eta to sample the Bernoulli variable thus determining whether there is an edge or not.  We have no a priori knowledge of \mu or \eta, except for a fully observed (features+edges) training graph with n vertices. In a partially observed (features only) test graph, we would like to infer where there are edges. We focus on a simplified test scenario, where two features (X1,X2) are sampled independently according to \mu, and an edge E is determined according to \eta(.|X1,X2). The learning goal is as follows: give the n-vertex training graph, construct a binary function g_n that maps two features into a label (edge present or absent). The performance of g is determined by the probability of error: R(g_n)=\Prob\{ g_n(X1,X2) \neq E \}.  If \eta is known, the Bayes (best) risk-achieving g^* can be obtained. To avoid overfitting, one would generally restrict the model of possible gs (say to VC dimension V), and then compare to the benchmark of the best-in-model, g'. The paper shows that g_n obtained by empirical risk minimization is only O(log(n)/n) off from g', in the following sense: there is a constant b, such that for all a > 1, w.p. > 1-d:  R(g_n)-R(g^*) \leq a (R(g')-R(g^*)) + (a+1/(a-1)) b V log(n/d)/n  This establishes an effective convergence rate of O(log(n)/n), improving on the previously best known rate of O(1/sqrt(n)).  The authors also propose a sub-sampling based method to scale the approach and give some numerical experiments to support the theory.  
- The paper is generally very well written. The result is interesting and solid, and potentially impactful. It is however a bit strange that the authors sell their results from the perspective of "fast rates despite no margin conditions" (presumably because of the proof techniques), while the crux of it is that "dependent samples are almost independent". This is because the number of (albeit not independent) samples is not n, but rather N=n(n-1)/2, and therefore we have a "slow" rate for N.  - When proposing the edge-sampling scaling technique, the authors claim that "sampling vertices directly would greatly damage the learning rate". The evidence that they give toward this is a variance comparison, on Line 63 in the supplementary material. First, the "is of order 1/m" claim on that line is not obvious (and probably incorrect). Second, if we forget sub-sampling and use (the even simpler) truncation method by discarding all but the first m vertices and the resulting sub-tree, then Theorem 1 applies as is, and we get an end-to-end rate of log(m)/m. This would correspond to B=m(m-1)/2 pairs, and thus we get the same rate of 1/sqrt(B) of Theorem 5 (without even the intermediary of the larger sample size n). Additionally, Table 1 (in both the main text and supplements) and Figure 1 (in the supplements) do not appear to show a difference in rate. Thus it would seem that the main "damage" is to the constants, not to the rate as claimed. The only other claim in this regard (Lines 244-246 in the supplementary material) is that the gap widens with increased model complexity. This wider gap could simply be due to the fact that both excess risks get multiplied by the VC dimension (so it could be a wider additive gap, while the multiplicative gap in excess risk remains constant).  Minor remarks:  - In Line 92, I think what is meant is "Marginalized over the features" and not "Conditioned upon the features". Once the features are specified, the edges are independently generated, though not identically. This should be stated plainly. On the other hand, of course the data point triplets Z_{ij}=(X_i,X_j,e_{ij}) are not independent (though marginally identically distributed).  - The term on Line 169, Lambda(g)=R(g)-R^*, may understandably be called excess risk. However, that terminology is often reserved for the excess over the best-in-model, rather than the Bayes risk. Especially given that the latter terminology is also used in the paper (Line 122), perhaps a different choice of words should be used on Line 169.