Convex Two-Layer Modeling with Latent Structure
Unsupervised learning of structured predictors has been a long standing pursuit in machine learning. Recently a conditional random field auto-encoder has been proposed in a two-layer setting, allowing latent structured representation to be automatically inferred. Aside from being nonconvex, it also requires the demanding inference of normalization. In this paper, we develop a convex relaxation of two-layer conditional model which captures latent structure and estimates model parameters, jointly and optimally. We further expand its applicability by resorting to a weaker form of inference-maximum a-posteriori. The flexibility of the model is demonstrated on two structures based on total unimodularity-graph matching and linear chain. Experimental results confirm the promise of the method.
A convex training formulation for a two-layer model p(z|y)p(y|x), i.e., the model Z <- Y <- X with X denoting the input, Y the latent space, and Z the output, is proposed and investigated. Contrasting existing work, the authors only require tractable MAP inference on p(y|x), which is available for graph-matching type models or generally any integer linear program with a totally unimodular constraint set.  A classical approach for two-layer models is the bi-level optimization based solution of p(z|\argmax_y p(y|x)). I.e., we first optimize the encoding model over y in an `inner' optimization, and infer the solution over the decoder in a second `outer' step. Training the parameters of those models is challenging due to the fact that small perturbations of the encoder model lead to the same prediction of the latent representation y, which is assumed to be discrete. Hence gradients for training the models are zero almost everywhere.  The presented approach plugs the first-order optimality conditions of the `inner' optimization into the `outer' optimization, by using Lagrange multipliers and a saddle-point formulation. The resulting problem is non-convex in the training parameters, and a semi-definite programming (SDP) formulation is proposed, which yields a convex relaxation after dropping the rank constraint.  Hence the authors state that learning depends on only the MAP operation over the latent space which is assumed to be efficient.  The proposed approach is demonstrated on the task of transliteration and image inpainting. The method outperforms the shown baselines. 
Review summary:  A nice technique to circumvent the issues arising from bi-level optimization is suggested and applied to probabilistic modeling. However quite a few details are missing to assess the quality of the proposed solution, e.g., additional assumptions beyond the given ones might be required (see point 1 below), important aspects are not explained (see point 3 below), and the experimental evaluation is a little weak. See below for details.  - Technical quality: derivations are sound but some additional assumptions should be required to ensure efficiency - Novelty: novel approach to two-layer modeling - Potential impact: hard to assess since details regarding efficiency of the proposed paper are not stated - Clarity: lacking explanation of some important aspects   I'll adjust my score based on the author feedback.  Review comments:  1. Let's assume z to be discrete. The decoder probability defined in Eq. (4) depends on the function G. To ensure proper normalization of the decoder distribution, G should correspond to the log-partition function. In the proposed formulation (e.g., Eq. (15)), maximization over its dual, i.e., the entropy restricted to the probability simplex, is required. Isn't tractability of this maximization task required/assumed then? In particular, the resulting program seems to be a quadratic term plus entropy terms and simplex constraints. Can the authors comment on how they solve this task efficiently? Sec. 5 does not seem to contain any details.  2. The following is not part of the proposed approach but the authors might nonetheless ensure correctness. In l.113, it is stated that the partition function `\Omega(Ux)' is replaced by an upper bound `\max -y'Ux'. The authors may want to check whether the maximization is indeed an `upper bound'. If I'm not wrong  \Omega(Ux) = \log\sum_y\exp(-y'Ux) = -y^\ast'UX + \log\sum_y\exp(-y'Ux + y^\ast'Ux) (*)  where y^\ast = \arg\max_y -y'Ux.  However the log term in (*) can either be positive or negative, hence the maximization is not really a bound. Did I overlook something?  3. Why are the model parameters bounded? There were no constraints in the original formulation illustrated in Eq. (7). Hence it's counterintuitive to see constraints in Eq. (10) and the following ones. I think any reader would appreciate a more sophisticated argument beyond the stated one: `to simplify the presentation'. Could the authors provide a few more details regarding the reason for introducing those bound constraints? Is feasibility an issue?  4. Can the authors comment on the quality of the proposed baselines? The method is compared to an approach from 2010 on the transliteration task. It might be useful to provide more recent baselines as well.  5. The authors may also want to comment on the computational complexity of the proposed method. Usage of an SDP formulation seems computationally expensive and not necessarily scalable to larger problem sizes.  6. Moreoever I'm wondering whether the authors compared the proposed SDP approach to gradient descent on the objective given in Eq. (15).  Minor comments:  - The authors could mention that the ^\prime operator denotes the transpose.