  Non-autoregressive (NAR) transformer models have been studied intensively in
automatic speech recognition (ASR), and a substantial part of NAR transformer
models is to use the casual mask to limit token dependencies. However, the
casual mask is designed for the left-to-right decoding process of the
non-parallel autoregressive (AR) transformer, which is inappropriate for the
parallel NAR transformer since it ignores the right-to-left contexts. Some
models are proposed to utilize right-to-left contexts with an extra decoder,
but these methods increase the model complexity. To tackle the above problems,
we propose a new non-autoregressive transformer with a unified bidirectional
decoder (NAT-UBD), which can simultaneously utilize left-to-right and
right-to-left contexts. However, direct use of bidirectional contexts will
cause information leakage, which means the decoder output can be affected by
the character information from the input of the same position. To avoid
information leakage, we propose a novel attention mask and modify vanilla
queries, keys, and values matrices for NAT-UBD. Experimental results verify
that NAT-UBD can achieve character error rates (CERs) of 5.0%/5.5% on the
Aishell1 dev/test sets, outperforming all previous NAR transformer models.
Moreover, NAT-UBD can run 49.8x faster than the AR transformer baseline when
decoding in a single step.

Computation and Language;Sound;Audio and Speech Processing