  For interpreting the behavior of a probabilistic model, it is useful to
measure a model's calibration--the extent to which it produces reliable
confidence scores. We address the open problem of calibration for tagging
models with sparse tagsets, and recommend strategies to measure and reduce
calibration error (CE) in such models. We show that several post-hoc
recalibration techniques all reduce calibration error across the marginal
distribution for two existing sequence taggers. Moreover, we propose tag
frequency grouping (TFG) as a way to measure calibration error in different
frequency bands. Further, recalibrating each group separately promotes a more
equitable reduction of calibration error across the tag frequency spectrum.

Computation and Language