A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
We propose a self-refining framework that enhances ASR performance with only
unlabeled datasets. The process starts with an existing ASR model generating
pseudo-labels on unannotated speech, which are then used to train a
high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs
are bootstrapped into the original ASR system, completing the closed-loop
self-improvement cycle. We demonstrated the effectiveness of the framework on
Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a
moderate amount of text data, and synthetic content from the AI models, we
adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error
rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching
benchmarks compared to Whisper. Results highlight the framework as a compelling
alternative to pseudo-labeling self-distillation approaches and provides a
practical pathway for improving ASR performance in low-resource or
domain-specific settings.
684fdd8e1d9b438aa39579c6
Cheng Kang Chou
684fdd8e1d9b438aa39579c7
Chan-Jan Hsu
684fdd8e1d9b438aa39579c8
Ho-Lam Chung
684fdd8e1d9b438aa39579c9
Liang-Hsuan Tseng
admin_assigned
2025-06-16T12:58:44.206Z
637b996e086af1cb122ead59
/avatars/c39d4c46d413b83cabf85940bd2423d1.svg
Liang-Hsuan Tseng
user
andybi7676
684fdd8e1d9b438aa39579ca
Hsi-Chun Cheng
claimed_verified
2025-06-18T12:17:02.752Z
64549cb802912fad3f1f4274
https://cdn-avatars.huggingface.co/v1/production/uploads/64549cb802912fad3f1f4274/M3qy1_xq_u3C0_-wtxsa7.png
Leo Cheng
user
lchengtw
684fdd8e1d9b438aa39579cb
Yu-Kuan Fu
admin_assigned
2025-06-16T12:58:50.206Z
64f21f1a73461a8c3abe43a5
/avatars/4b7ff46ae12476f5b6727bd5cc303f38.svg
Yu-Kuan Fu
user
YuKuanFu
684fdd8e1d9b438aa39579cc
Kuan Po Huang
684fdd8e1d9b438aa39579cd
Hung-Yi Lee
admin_assigned
2025-06-16T12:59:12.434Z
629b6eb3ad498ab19282ae6f
/avatars/adade8c512122120b06e5c50724a178d.svg
Hung-yi Lee
user
hungyilee
A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
We propose a self-refining framework that enhances ASR performance with only
unlabeled datasets. The process starts with an existing ASR model generating
pseudo-labels on unannotated speech, which are then used to train a
high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs
are bootstrapped into the original ASR system, completing the closed-loop
self-improvement cycle. We demonstrated the effectiveness of the framework on
Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a
moderate amount of text data, and synthetic content from the AI models, we
adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error
rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching
benchmarks compared to Whisper. Results highlight the framework as a compelling
alternative to pseudo-labeling self-distillation approaches and provides a
practical pathway for improving ASR performance in low-resource or
domain-specific settings.
A self-refining framework enhances ASR performance using unlabeled datasets by integrating pseudo-labeling, TTS, and synthesized speech to create a specialized model.
self-refining framework ASR pseudo-labels TTS synthesized speech Whisper-large-v2 Twister error rates Mandarin Mandarin-English code-switching