A Smoother Way to Train Structured Prediction Models
We present a framework to train a structured prediction model by performing smoothing on the inference algorithm it builds upon. Smoothing overcomes the non-smoothness inherent to the maximum margin structured prediction objective, and paves the way for the use of fast primal gradient-based optimization algorithms. We illustrate the proposed framework by developing a novel primal incremental optimization algorithm for the structural support vector machine. The proposed algorithm blends an extrapolation scheme for acceleration and an adaptive smoothing scheme and builds upon the stochastic variance-reduced gradient algorithm. We establish its worst-case global complexity bound and study several practical variants. We present experimental results on two real-world problems, namely named entity recognition and visual object localization. The experimental results show that the proposed framework allows us to build upon efficient inference algorithms to develop large-scale optimization algorithms for structured prediction which can achieve competitive performance on the two real-world problems.
Overview: This paper proposes an accelerated variance-reduction algorithm for training structured predictors. In this approach the training objective is augmented with a proximal term anchored with a momentum point (eq (3)), the loss is smoothed using Nesterov's smoothing method (adding entropy or L2 to the dual), and a linear-rate solver (SVRG) is applied to the resulting objective in the inner loop. This achieves accelerated convergence rates for training.   Comments: * I think that the connection to structured prediction is somewhat weak. In particular, the analysis uses the finite sum and smoothability of the training objective. The first is not specific to structured prediction, and the latter has been studied extensively in previous work, including for learning (see next point). Inference is assumed to be an atomic operation performed via calls to an inference oracle -- assuming access to efficient top-k and expectations computation, and there is no leveraging of instance-level structure.  * I don't think it's fair to say that smooth inferene oracles are a contribution of this paper (line 40). Entropic smoothing was used much earlier, for example: - Convex Relaxation Methods for Graphical Models: Lagrangian and Maximum Entropy Approaches, Johnson 2008. - Fast and smooth: Accelerated dual decomposition for MAP inference, Jojic et al. 2010. - [Adaptive smoothing] Efficient MRF Energy Minimization via Adaptive Diminishing Smoothing, Savchynskyy et al. 2012. Also in the context of learning (SSVMs), e.g.: - Blending Learning and Inference in Conditional Random Fields; Hazan et al. JMLR 2016. L2 smoothing for inference was used more recently, for example: - Smooth and Strong: MAP Inference with Linear Convergence, Meshi et al. 2015. - [For deep structured models] From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification, Martins and Astudillo 2016.  * Top-K oracles may sometimes be more expensive than max oracles (also true for marginal oracles). It’s worth providing more details on that to clarify when it’s not a good idea to use this type of smoothing.  * In the experiments (Fig 2), the accelerated methods don't seem to have an advantage over plain SVRG in terms of test performance.   Minor comments: * Lines 62-63: the remark about the loss being decomposable is not really necessary since nothing is assumed about the score (line 58). Instead, perhaps better to directly extend the assumption on the *oracle* to include (2). Otherwise, score decomposition may be mentioned.  * Line 81: it would be good to clarify that u_2 depends on mu via notation, e.g., u_2(y;\mu) or u_\mu(y).  * Line 194 [typo]: 'set reduced set'.  ================= After author response: To better place this work in context of existing literature on structured prediction, I suggest you down-tone the contribution on smoothing of inference and cite a few works on Nesterov smoothing of inference objectives. I also think you should add a discussion on complexity of the required oracles making it clear that smoothing often requires more expensive oracles.