Calibrated Structured Prediction
In user-facing applications, displaying calibrated confidence measuresprobabilities that correspond to true frequency-can be as important as obtaining high accuracy. We are interested in calibration for structured prediction problems such as speech recognition, optical character recognition, and medical diagnosis. Structured prediction presents new challenges for calibration: the output space is large, and users may issue many types of probability queries (e.g., marginals) on the structured output. We extend the notion of calibration so as to handle various subtleties pertaining to the structured setting, and then provide a simple recalibration method that trains a binary classifier to predict probabilities of interest. We explore a range of features appropriate for structured recalibration, and demonstrate their efficacy on three real-world datasets.
This paper proposes a method to calibrate structured predictions by minimizing the l_2 loss between true outcome and predictive probability.The approach "re-maps" an existing structured prediction model's predictive distribution for each example to be more calibrated.I am somewhat skeptical and confused by the motives of this work.Calibration seems like a sensible concern for weather forecasts because the user-facing "output" of the forecast is the probability prediction itself.Though the predictions from speech recognition, OCR, scene labeling, etc. support "user-facing" applications, the "output" is not the probability estimate for prediction, but the MAP estimate: i.e., words, labeled scenes, detected faces, etc. Should a user feel more satisfied that their structured predictor is more internally consistent (in terms of calibration) when the end-use application performance (e.g., accuracy) is worse as a result?Can a compelling structured prediction application where calibration concerns seem justified be provided?"More often, the probabilities obtained ... will not be useful in practice, mainly because they will not be calibrated" (line 081)It is absurd to suggest that all of the non-calibrated structured prediction methods currently used in practice are not useful in practice.Figure 1b: the example probabilities do not appear consistent P("land") + P("lano") < P("l***").Much of the description of background work is light on references (in fact, there are zero references in the "Background" section!) and the connections to previous work not fully illuminated.For example, the l_2 loss is also known as the Brier loss and has been fairly extensively studied.The components of decomposition in (3) were originally called uncertainty, tresolution, and reliability [13].It is unclear where e.g., the entropy term comes from -- it does not seem to match with, e.g., the Shannon entropy.The experiments show that the calibration method proposed does indeed provide better calibration than the original structured predictions.The small comparison between decision tree-based recalibration and k-NN-based recalibration seems insufficient.If this idea of nearest neighbor re-calibration is novel, why not provide comparisons also in the classification setting with [15, 18, 14]?Significantly more demonstration and/or discussion of why these previous methods cannot also be applied in some manner to these experiments is warranted.-------------The author feedback argues that quantifying prediction uncertainty is important, which I agree with, but this still doesn't explain why _calibrated_ uncertainty is essential.The authors seem to equate the two, but there are many models that estimate uncertainty without being well-calibrated.The reference provided to a structured prediction application in bioinformatics, "Taxonomic metagenome sequence assignment with structured output models," uses SVM-based methods presumably to optimize accuracy rather than calibrated uncertainty. I'm not sure how this reference is supposed to advance the argument.