What do self-supervised speech models know about Dutch? Analyzing
  advantages of language-specific pre-training
How language-specific are speech representations learned by self-supervised
models? Existing work has shown that a range of linguistic features can be
successfully decoded from end-to-end models trained only on speech recordings.
However, it's less clear to what extent pre-training on specific languages
improves language-specific linguistic information. Here we test the encoding of
Dutch phonetic and lexical information in internal representations of
self-supervised Wav2Vec2 models. Pre-training exclusively on Dutch improves the
representation of Dutch linguistic features as compared to pre-training on
similar amounts of English or larger amounts of multilingual data. This
language-specific advantage is well-detected by trained clustering or
classification probes, and partially observable using zero-shot metrics.
Furthermore, the language-specific benefit on linguistic feature encoding
aligns with downstream performance on Automatic Speech Recognition.
6842d09f619c27947a4a76cd
Marianne de Heer Kloots
claimed_verified
2025-06-07T05:48:31.333Z
63da7c56e697e5898cbba61c
/avatars/2385d83f188644f65f7f474c35ae72cc.svg
Marianne de Heer Kloots
user
mariannedhk
6842d09f619c27947a4a76ce
Hosein Mohebbi
6842d09f619c27947a4a76cf
Charlotte Pouw
6842d09f619c27947a4a76d0
Gaofei Shen
6842d09f619c27947a4a76d1
Willem Zuidema
6842d09f619c27947a4a76d2
Martijn Bentum
What do self-supervised speech models know about Dutch? Analyzing
  advantages of language-specific pre-training
How language-specific are speech representations learned by self-supervised
models? Existing work has shown that a range of linguistic features can be
successfully decoded from end-to-end models trained only on speech recordings.
However, it's less clear to what extent pre-training on specific languages
improves language-specific linguistic information. Here we test the encoding of
Dutch phonetic and lexical information in internal representations of
self-supervised Wav2Vec2 models. Pre-training exclusively on Dutch improves the
representation of Dutch linguistic features as compared to pre-training on
similar amounts of English or larger amounts of multilingual data. This
language-specific advantage is well-detected by trained clustering or
classification probes, and partially observable using zero-shot metrics.
Furthermore, the language-specific benefit on linguistic feature encoding
aligns with downstream performance on Automatic Speech Recognition.
Self-supervised Wav2Vec2 models encode Dutch linguistic features more accurately when pre-trained exclusively on Dutch data, compared to similar amounts of English or multilingual data, as shown by clustering and classification probes, and demonstrated through improved Automatic Speech Recognition performance.
Wav2Vec2 phonetic information lexical information self-supervised models clustering probes classification probes Automatic Speech Recognition