Stochastic Structured Prediction under Bandit Feedback
Stochastic structured prediction under bandit feedback follows a learning protocol where on each of a sequence of iterations, the learner receives an input, predicts an output structure, and receives partial feedback in form of a task loss evaluation of the predicted structure. We present applications of this learning scenario to convex and non-convex objectives for structured prediction and analyze them as stochastic first-order methods. We present an experimental evaluation on problems of natural language processing over exponential output spaces, and compare convergence speed across different objectives under the practical criterion of optimal task performance on development data and the optimization-theoretic criterion of minimal squared gradient norm. Best results under both criteria are obtained for a non-convex objective for pairwise preference learning under bandit feedback.
This paper applies stochastic approximation methods in training log linear probabilistic structured prediction models under the bandit setting. It differs from the full information setting in that the learner only receives partial feedback for every prediction in the form of task loss evaluation, rather than a ground truth output with which to validate its prediction. The author(s)' stated main contribution of the paper is to apply stochastic first order (SFO) methods to learning with pairwise comparison feedback, and showing that it achieves the fastest convergence over two other settings (loss and cross-entropy minimization) based on empirical evaluations. The convergence properties of all three methods are analyzed based on existing results in the literature, which assume smoothness and boundedness conditions. Despite worst-case theoretical guarantees that favor the strongly convex cross-entropy objective, the author(s) reason that pairwise learning performed better due to smaller constant factors such as the variance of stochastic gradient updates. This conclusion is drawn from experimental evaluations.
A main result of this paper is that under bandit feedback, applying SFO methods in pairwise preference ranking (PR) and expected loss minimization (EL) can lead to better empirical convergence rates than in cross-entropy (CE) minimization, despite theoretical results that might suggest otherwise. The author(s) explained this result by estimating the constant factors of the bounds, such as the variance of stochastic updates, showing that they are orders of magnitude smaller for PR and CE. This leads to a statement in the conclusion that '... constants such as the variance of the stochastic gradient update can offset possible theoretical gains in convergence speed.'  Applying SFO methods in PR is interesting and has potential applications, but I see several issues with drawing the conclusion above from empirical evaluations, 1. The learning rates for all three methods were fixed at 1e^-6: the authors stated in line 183-184 that theoretical convergence for strongly convex functions (which applies to CE) were analyzed under diminishing learning rates, but used a constant rate in the experiments. I am not aware of similar bounds that hold under constant learning rate in the strongly convex case. How can one be certain that the apparent advantages of PR and EL over CE are not due to suboptimal choices of learning rates? 2. There is no justification of why 1e^-6 is chosen. Does the conclusion still hold under different values? 3. The constant factor estimates are based on averaging the results of 3 runs of each algorithm; I think the sample may be too small.  Another issue of this paper is that a number statements were presented without proper justification or context, which makes it difficult to assess their correctness. This is problematic because several of them are crucial for the paper's main results to hold. For example, 1. line 158-159, 'Note that multiplying the gradient by ...' -- Isn't this multiplication by 1? It is not clear how the sampling is done. 2. line 159-162, 'While minimization of this objective ... This further increases the variance ...' -- This explanation is unclear to me. Since this relates to a central claim of the paper, it would be helpful to clarify it. 3. line 175-177 -- These assumptions are crucial and it would help to be more precise here, e.g., by deriving the bounds on the variance explicitly under all 3 objectives. 4. line 187-189, For constant learning rates, even faster convergence can be shown ... -- What is the implication of this and how is 'faster' characterized? 5. line 193-194, 'Second, the standard asymptotic complexity bounds hide the constants ...' -- It would be helpful to state the bounds precisely with the inclusion of epsilon and all these constants, along with any other factors that may affect convergence rates (e.g, dimension of w and learning rate choices).  It would also help to provide more context in Related Work section. Instead of generic references to reinforcement learning and bandit learning (which are too broad), the author(s) could survey existing approaches to probabilistic structured prediction, going into more depth on how SA methods have been applied to closely related problems, and what this paper does differently.  Overall, I find the evidence presented in this paper and its explanations inadequate to support the conclusions. The paper could also use improvements in clarity and rigor of writing, especially when presenting theoretical results of convergence rates and the factors that influence them. For these reasons, I am unable to recommend it for publication.