Spectral Norm Regularization of Orthonormal Representations for Graph Transduction
Recent literature [1] suggests that embedding a graph on an unit sphere leads to better generalization for graph transduction. However, the choice of optimal embedding and an efficient algorithm to compute the same remains open. In this paper, we show that orthonormal representations, a class of unit-sphere graph embeddings are PAC learnable. Existing PAC-based analysis do not apply as the VC dimension of the function class is infinite. We propose an alternative PAC-based bound, which do not depend on the VC dimension of the underlying function class, but is related to the famous Lovász ϑ function. The main contribution of the paper is SPORE, a SPectral regularized ORthonormal Embedding for graph transduction, derived from the PAC bound. SPORE is posed as a non-smooth convex function over an elliptope. These problems are usually solved as semi-definite programs (SDPs) with time complexity O(n 6 ). We present, Infeasible Inexact proximal (IIP): an Inexact proximal method which performs subgradient procedure on an approximate projection, not necessarily feasible. IIP is more scalable than SDP, has an O(
) convergence, and is generally applicable whenever a suitable approximate projection is available. We use IIP to compute SPORE where the approximate projection step is computed by FISTA, an accelerated gradient descent procedure. We show that the method has a convergence rate of O(
). The proposed algorithm easily scales to 1000's of vertices, while the standard SDP computation does not scale beyond few hundred vertices. Furthermore, the analysis presented here easily extends to the multiple graph setting.
The paper starts by introducing the background and showing the contributions. The authors then use the Zhang and Ando result showing that SSL reduces to an equivalent kernel-based supervised learning problem for the rest of the paper. In section 2 and appendix, they provide an error bound showing that minimising the spectral norm of the graph kernel matrix is a good way to improve generalisation. In section 3, they add a spectral norm regularization term to the Zhang and Ando formulation, and provide links between the error convergence and the Lovasz number of the data graph. Section 4 proposes a proximal solver, where the specificity is that it deals with the projection step approximately to handle the constraint of a cone-polytope intersection. Section 5 proposes an MKL extension, and section 6 concludes with experimental results on several datasets.            Overall, the paper is interesting and contains a lot of material (maybe even a bit dense for a single paper), with several important contributions. The link between large-scale topological properties of the data graph (Lovasz number), which reflect its organisation in some space or manifold, and the convergence of the embedding estimation (eq. 5) is particularly enlightening and bridges several fields.            In section 1 after equation 3 I would perhaps make explicit that the correspondance between eqs. (3) and (1) means that a semi-supervised learning problem becomes a supervised learning problem, so that the learning is on labelled samples, otherwise it is a bit surprising to see indices over S only (with no \bar{S}) in the sums in equation (3), whereas in eq. 1 the regulariser is also on non-labeled samples while the loss is on labelled data.      In section 2, The notation of equation 5 is confusing - why is there an equal sign just before \omega_C ?            In section 3 around line 162, maybe mention how the embedding (U) is recovered from K ? There are several (practical) choices for the factorisation.            In section 4,Algorithm 1 step 8-9 - ensuring that K_r is PSD typically quite expensive (for general dense square symmetric matrices), and here an eigendecomposition is done every time (supplementary D.3.2). Is this where the algorithm takes most time? Would it be feasible to just optimise on a factorisation of K (say, Cholesky with K=LL^T), projection to the feasible set would be to a matrix with all-zeros in upper triangular, which would by construction ensure that K is PSD ?            Zhang and Ando is in NIPS 2005, not 2006