BUT System for the MLC-SLM Challenge
We present a two-speaker automatic speech recognition (ASR) system that
combines DiCoW -- a diarization-conditioned variant of Whisper -- with
DiariZen, a diarization pipeline built on top of Pyannote. We first evaluate
both systems in out-of-domain (OOD) multilingual scenarios without any
fine-tuning. In this scenario, DiariZen consistently outperforms the baseline
Pyannote diarization model, demonstrating strong generalization. Despite being
fine-tuned on English-only data for target-speaker ASR, DiCoW retains solid
multilingual performance, indicating that encoder modifications preserve
Whisper's multilingual capabilities. We then fine-tune both DiCoW and DiariZen
on the MLC-SLM challenge data. The fine-tuned DiariZen continues to outperform
the fine-tuned Pyannote baseline, while DiCoW sees further gains from domain
adaptation. Our final system achieves a micro-average tcpWER/CER of 16.75% and
ranks second in Task 2 of the MLC-SLM challenge. Lastly, we identify several
labeling inconsistencies in the training data -- such as missing speech
segments and incorrect silence annotations -- which can hinder diarization
fine-tuning. We propose simple mitigation strategies to address these issues
and improve system robustness.
6853f62841586a7b0193a0ba
Alexander Polok
6853f62841586a7b0193a0bb
Jiangyu Han
6853f62841586a7b0193a0bc
Dominik Klement
6853f62841586a7b0193a0bd
Samuele Cornell
6853f62841586a7b0193a0be
Jan Černocký
6853f62841586a7b0193a0bf
Lukáš Burget
BUT System for the MLC-SLM Challenge
We present a two-speaker automatic speech recognition (ASR) system that
combines DiCoW -- a diarization-conditioned variant of Whisper -- with
DiariZen, a diarization pipeline built on top of Pyannote. We first evaluate
both systems in out-of-domain (OOD) multilingual scenarios without any
fine-tuning. In this scenario, DiariZen consistently outperforms the baseline
Pyannote diarization model, demonstrating strong generalization. Despite being
fine-tuned on English-only data for target-speaker ASR, DiCoW retains solid
multilingual performance, indicating that encoder modifications preserve
Whisper's multilingual capabilities. We then fine-tune both DiCoW and DiariZen
on the MLC-SLM challenge data. The fine-tuned DiariZen continues to outperform
the fine-tuned Pyannote baseline, while DiCoW sees further gains from domain
adaptation. Our final system achieves a micro-average tcpWER/CER of 16.75% and
ranks second in Task 2 of the MLC-SLM challenge. Lastly, we identify several
labeling inconsistencies in the training data -- such as missing speech
segments and incorrect silence annotations -- which can hinder diarization
fine-tuning. We propose simple mitigation strategies to address these issues
and improve system robustness.
The combined DiCoW and DiariZen ASR system demonstrates strong performance in multilingual scenarios, with DiCoW preserving its multilingual capabilities and DiariZen improving through fine-tuning.
DiCoW DiariZen Whisper Pyannote ASR out-of-domain multilingual fine-tuning micro-average tcpWER/CER MLC-SLM challenge labeling inconsistencies