Learning latent variable structured prediction models with Gaussian perturbations
The standard margin-based structured prediction commonly uses a maximum loss over all possible structured outputs [26,1,5,25]. The large-margin formulation including latent variables [30,21] not only results in a non-convex formulation but also increases the search space by a factor of the size of the latent space. Recent work [11] has proposed the use of the maximum loss over random structured outputs sampled independently from some proposal distribution, with theoretical guarantees. We extend this work by including latent variables. We study a new family of loss functions under Gaussian perturbations and analyze the effect of the latent space on the generalization bounds. We show that the non-convexity of learning with latent variables originates naturally, as it relates to a tight upper bound of the Gibbs decoder distortion with respect to the latent space. Finally, we provide a formulation using random samples and relaxations that produces a tighter upper bound of the Gibbs decoder distortion up to a statistical accuracy, which enables a polynomial time evaluation of the objective function. We illustrate the method with synthetic experiments and a computer vision application.
UPDATE:  Upon reading the author response, I have decided to leave my review unchanged.  The topic of this work is of interest to the community, and it can make a decent publication. I am still concerned about novelty. While the author response argued that similar increments have been done in past works, I feel that there are some significant differences between the current case and the mentioned works. Accepting the paper is still a reasonable decision in my humble opinion.  Original Review:  The paper develops PAC-Bayesian bounds for margin-based structured prediction, when hidden variables are included in the problem. It mainly relies on the analysis of [1], adapted to the formulation of structured SVM with latent variables given in [2].  The point I found most interesting is the remark about attempts to convexify structured losses studied in the paper. The necessity for a non-convex formulation stems directly from the PAC-Bayesian bound, which is a nice and novel insight.  My main concern regarding this paper is whether it is sufficiently novel. The majority of technical work is an almost direct adaptation of [1], and the paper does not seem to offer a significantly novel approach for the analysis of structured prediction problems. With that said, the problem of latent variables in structured prediction is interesting. This is enough to shift my overall score to 6.  The paper could also gain from a more thorough experimental study. In all the proposed experiments, the latent space is very small and marginalization or enumeration of the hidden states is easy. I think that a problem with a large latent space, where the randomization of this space gives a significant advantage, could strengthen the point of the paper. This would also distinguish the algorithmic contribution of this paper from the algorithms analyzed in [1] and demonstrated in papers on natural language processing.  Overall, I think that while this paper makes nice contributions, it is not clear if they amount to a significantly novel submission.  [1] Honorio, J. and Jaakkola, T. Structured prediction: from Gaussian perturbations to linear-time principled algorithms. UAI 2016 [2] Yu, C. and Joachims, T. Learning structural SVMs with latent variables. ICML 2009