Label-Looping: Highly Efficient Decoding for Transducers
<|reference_start|>Label-Looping: Highly Efficient Decoding for Transducers: This paper introduces a highly efficient greedy decoding algorithm for Transducer-based speech recognition models. We redesign the standard nested-loop design for RNN-T decoding, swapping loops over frames and labels: the outer loop iterates over labels, while the inner loop iterates over frames searching for the next non-blank symbol. Additionally, we represent partial hypotheses in a special structure using CUDA tensors, supporting parallelized hypotheses manipulations. Experiments show that the label-looping algorithm is up to 2.0X faster than conventional batched decoding when using batch size 32. It can be further combined with other compiler or GPU call-related techniques to achieve even more speedup. Our algorithm is general-purpose and can work with both conventional Transducers and Token-and-Duration Transducers. We open-source our implementation to benefit the research community.<|reference_end|>
@article{bataev2024label-looping:,
    title={Label-Looping: Highly Efficient Decoding for Transducers},
    author={Vladimir Bataev, Hainan Xu, Daniel Galvez, Vitaly Lavrukhin, Boris
  Ginsburg},
    journal={arXiv preprint arXiv:2406.06220},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06220},
    primaryClass={eess.AS cs.AI cs.CL cs.LG cs.SD}
}