UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models
<|reference_start|>UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models: Non-autoregressive automatic speech recognition (NASR) models have gained attention due to their parallelism and fast inference. The encoder-based NASR, e.g. connectionist temporal classification (CTC), can be initialized from the speech foundation models (SFM) but does not account for any dependencies among intermediate tokens. The encoder-decoder-based NASR, like CTC alignment-based single-step non-autoregressive transformer (CASS-NAT), can mitigate the dependency problem but is not able to efficiently integrate SFM. Inspired by the success of recent work of speech-text joint pre-training with a shared transformer encoder, we propose a new encoder-based NASR, UniEnc-CASSNAT, to combine the advantages of CTC and CASS-NAT. UniEnc-CASSNAT consists of only an encoder as the major module, which can be the SFM. The encoder plays the role of both the CASS-NAT encoder and decoder by two forward passes. The first pass of the encoder accepts the speech signal as input, while the concatenation of the speech signal and the token-level acoustic embedding is used as the input for the second pass. Examined on the Librispeech 100h, MyST, and Aishell1 datasets, the proposed UniEnc-CASSNAT achieves state-of-the-art NASR results and is better or comparable to CASS-NAT with only an encoder and hence, fewer model parameters. Our codes are publicly available.<|reference_end|>
@article{fan2024unienc-cassnat:,
    title={UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL
  Models},
    author={Ruchao Fan, Natarajan Balaji Shanka, and Abeer Alwan},
    journal={arXiv preprint arXiv:2402.08898},
    year={2024},
    doi={10.1109/LSP.2024.3365036},
    archivePrefix={arXiv},
    eprint={2402.08898},
    primaryClass={eess.AS cs.CL cs.SD}
}