  Phones and their context-dependent variants have been the standard modeling
units for conventional speech recognition systems, while characters and
subwords have demonstrated their effectiveness for end-to-end recognition
systems. We investigate the use of phone-based subwords, in particular, byte
pair encoder (BPE), as modeling units for end-to-end speech recognition. In
addition, we also developed multi-level language model-based decoding
algorithms based on a pronunciation dictionary. Besides the use of the lexicon,
which is easily available, our system avoids the need of additional expert
knowledge or processing steps from conventional systems. Experimental results
show that phone-based BPEs tend to yield more accurate recognition systems than
the character-based counterpart. In addition, further improvement can be
obtained with a novel one-pass joint beam search decoder, which efficiently
combines phone- and character-based BPE systems. For Switchboard, our
phone-based BPE system achieves 6.8\%/14.4\% word error rate (WER) on the
Switchboard/CallHome portion of the test set while joint decoding achieves
6.3\%/13.3\% WER. On Fisher + Switchboard, joint decoding leads to 4.9\%/9.5\%
WER, setting new milestones for telephony speech recognition.

Audio and Speech Processing