  Recent developments in Named Entity Recognition (NER) have resulted in better
and better models. However, is there a glass ceiling? Do we know which types of
errors are still hard or even impossible to correct? In this paper, we present
a detailed analysis of the types of errors in state-of-the-art machine learning
(ML) methods. Our study reveals the weak and strong points of the Stanford,
CMU, FLAIR, ELMO and BERT models, as well as their shared limitations. We also
introduce new techniques for improving annotation, for training processes and
for checking a model's quality and stability. Presented results are based on
the CoNLL 2003 data set for the English language. A new enriched semantic
annotation of errors for this data set and new diagnostic data sets are
attached in the supplementary materials.

Computation and Language