  In this work, we present Lexical Unit Analysis (LUA), a framework for general
sequence segmentation tasks. Given a natural language sentence, LUA scores all
the valid segmentation candidates and utilizes dynamic programming (DP) to
extract the maximum scoring one. LUA enjoys a number of appealing properties
such as inherently guaranteeing the predicted segmentation to be valid and
facilitating globally optimal training and inference. Besides, the practical
time complexity of LUA can be reduced to linear time, which is very efficient.
We have conducted extensive experiments on 5 tasks, including syntactic
chunking, named entity recognition (NER), slot filling, Chinese word
segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our
models have achieved the state-of-the-art performances on 13 of them. The
results also show that the F1 score of identifying long-length segments is
notably improved.

Computation and Language