Convex Two-Layer Modeling
Latent variable prediction models, such as multi-layer networks, impose auxiliary latent variables between inputs and outputs to allow automatic inference of implicit features useful for prediction. Unfortunately, such models are difficult to train because inference over latent variables must be performed concurrently with parameter optimization-creating a highly non-convex problem. Instead of proposing another local training method, we develop a convex relaxation of hidden-layer conditional models that admits global training. Our approach extends current convex modeling approaches to handle two nested nonlinearities separated by a non-trivial adaptive latent layer. The resulting methods are able to acquire two-layer models that cannot be represented by any single-layer model over the same features, while improving training quality over local heuristics.
The paper addresses problem of training two-layer 
      network for classification with latent variables. The authors propose a 
      convex SDP relaxation of originally non-convex training problem. They also 
      provide an approximate optimization algorithm to solve their SDP 
      formulation. A proof of concept experiments show promising results, 
      namely, that the algorithm outperforms both globally optimized 
      single-layer models as well as the same two-layer model optimized with 
      local alternating minimization. The proposed reformulation which 
      allows SDP relaxation is interesting and novel. Overall the paper is 
      sufficiently clear though some parts of the text are dense. The paper 
      seems to be technically sound. The main weakness seems to be 
      complexity of the resulting SDP problem (21). The authors could 
      mention basic properties of the proposed optimization algorithm for 
      solving (21), e.g computational time required for the benchmark problems 
      and whether the algorithm provides a precise solution (i.e. what was 
      the stopping condition used). This is an important information because 
      convex problem does not immediately mean easy to solve problem, i.e. a 
      convex relaxation can be intractable in practice and it should be 
      clear if it is the case or not. However, the proposed relaxation would be 
      valuable even in this case. Minor comments: - equ (17): 
      variable is missing below the first minimum - equ (18): I think 
      N=\Phi'*\Phi should appear in the conditions defining the set. - line 
      331: (d) Synhetic results - line 356: It is unclear why the used 
      transductive evaluation does not require computing responses of the 
      network and thus knowledge of W and V. - I could not find the actual 
      size of instances of the problem (21) solved in the experiments.