CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions
<|reference_start|>CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions: We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder's cross-attention scores. We fine-tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verbatim speech transcription, word segmentation, and the timed detection of filler events, and can further mitigate transcription hallucinations. The code is available open https://github.com/nyrahealth/CrisperWhisper.<|reference_end|>
@article{wagner2024crisperwhisper:,
    title={CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions},
    author={Laurin Wagner, Bernhard Thallinger, Mario Zusag},
    journal={arXiv preprint arXiv:2408.16589},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16589},
    primaryClass={cs.LG}
}