  This paper presents new state-of-the-art models for three tasks,
part-of-speech tagging, syntactic parsing, and semantic parsing, using the
cutting-edge contextualized embedding framework known as BERT. For each task,
we first replicate and simplify the current state-of-the-art approach to
enhance its model efficiency. We then evaluate our simplified approaches on
those three tasks using token embeddings generated by BERT. 12 datasets in both
English and Chinese are used for our experiments. The BERT models outperform
the previously best-performing models by 2.5% on average (7.5% for the most
significant case). Moreover, an in-depth analysis on the impact of BERT
embeddings is provided using self-attention, which helps understanding in this
rich yet representation. All models and source codes are available in public so
that researchers can improve upon and utilize them to establish strong
baselines for the next decade.

Computation and Language