--- datasets: - CoRal-project/coral-v2 language: - da base_model: - facebook/wav2vec2-xls-r-300m metrics: - wer - cer license: openrail pipeline_tag: automatic-speech-recognition model-index: - name: roest-wav2vec2-315m-v2 results: - task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: CoRal read-aloud type: alexandrainst/coral split: test args: read_aloud metrics: - type: cer value: 6.5% ± 0.2% name: CER - type: wer value: 16.3% ± 0.4% name: WER --- # Pre-release of Roest-wav2vec2-315m-v2 This is a pre-release of a Danish state-of-the-art speech recognition model, trained as part of the CoRal project by [Alvenir](https://www.alvenir.ai/). This repository contains a Wav2Vec2 model trained on the [CoRal-v2 dataset](https://huggingface.co/datasets/CoRal-project/coral-v2/tree/main). The CoRal-v2 dataset includes a rich variety of Danish conversational and read-aloud data, distributed across diverse age groups, genders, and dialects. The model is designed for automatic speech recognition (ASR). ## Quick Start Start by installing the required libraries: ```shell $ pip install transformers kenlm pyctcdecode ``` Next you can use the model using the `transformers` Python package as follows: ```python >>> from transformers import pipeline >>> audio = get_audio() # 16kHz raw audio array >>> transcriber = pipeline(model="CoRal-project/roest-wav2vec2-315m-v2") >>> transcriber(audio) {'text': 'your transcription'} ``` --- ## Transcription Examples Explore the following audio samples along with their transcriptions and accuracy metrics. Each example showcases the model's performance with different Danish dialects. ### Example 1 - Vestjysk Dialect **Audio Sample:** **Model Transcription:** *det blev til yderlig ti mål i den første sæson på trods af en position som back* **Target Transcription:** *det blev til yderligere ti mål i den første sæson på trods af en position som back* - **Character Error Rate (CER):** 3.7% - **Word Error Rate (WER):** 5.9% --- ### Example 2 - Sønderjysk Dialect **Audio Sample:** **Model Transcription:** *en arkitektoniske udformning af pladser forslagene iver benzen* **Target Transcription:** *den arkitektoniske udformning af pladsen er forestået af ivar bentsen* - **Character Error Rate (CER):** 20.3% - **Word Error Rate (WER):** 60.0% --- ### Example 3 - Nordsjællandsk Dialect **Audio Sample:** **Model Transcription:** *østrig og ungarn samarbejder om søen gennem den østrigske og ungarske vandkommission* **Target Transcription:** *østrig og ungarn samarbejder om søen gennem den østrigske og ungarske vandkommission* - **Character Error Rate (CER):** 0.0% - **Word Error Rate (WER):** 0.0% --- ### Example 4 - Lollandsk Dialect **Audio Sample:** **Model Transcription:** *det er produceret af thomas helme og indspillede i easy sound recording studio i københavn* **Target Transcription:** *det er produceret af thomas helmig og indspillet i easy sound recording studio i københavn* - **Character Error Rate (CER):** 4.4% - **Word Error Rate (WER):** 13.3% --- ## Model Details Wav2Vec2 is a state-of-the-art model architecture for speech recognition, leveraging self-supervised learning from raw audio data. The pre-trained [Wav2Vec2-XLS-R-300M](facebook/wav2vec2-xls-r-300m) has been fine-tuned for automatic speech recognition with the [CoRal-v2 dataset](https://huggingface.co/datasets/CoRal-project/coral-v2/tree/main) dataset to enhance its performance in recognizing Danish speech with consideration to different dialects. The model was trained for 30K steps using the training setup in the [CoRaL repository](https://github.com/alexandrainst/coral/tree) by running: ``` python src/scripts/finetune_asr_model.py model=wav2vec2-small max_steps=30000 datasets.coral_conversation_internal.id=CoRal-project/coral-v2 datasets.coral_readaloud_internal.id=CoRal-project/coral-v2 ``` The model is evaluated using a Language Model (LM) as post-processing. The utilized LM is the one trained and used by [CoRal-project/roest-wav2vec2-315m-v1](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v2). --- ## Dataset ### [CoRal-v2](https://huggingface.co/datasets/CoRal-project/coral-v2/tree/main) - **Subsets**: - Conversation - Read-aloud - **Language**: Danish. - **Variation**: Includes various dialects, age groups, and gender distinctions. ### License Note that the dataset used is licensed under a custom license, adapted from OpenRAIL-M, which allows commercial use with a few restrictions (speech synthesis and biometric identification). See [license](https://huggingface.co/Alvenir/coral-1-whisper-large/blob/main/LICENSE). --- ## Evaluation The model was evaluated using the following metrics: - **Word Error Rate (WER)**: The percentage of words incorrectly transcribed. - **Character Error Rate (CER)**: The percentage of characters incorrectly transcribed. **OBS!** It should be noted that the [CoRal test dataset](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) does not contain any conversation data, whereas the model is trained for read-aloud and conversation, but is only tested on read-aloud in the [CoRal test dataset](https://huggingface.co/datasets/CoRal-project/coral/viewer/read_aloud/test). | Model | Number of parameters | Finetuned on data of type | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) CER | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) WER | | :----------------------------------------------------------------------------------------------- | -------------------: | --------------------------: | --------------------------------------------------------------------------------------: | --------------------------------------------------------------------------------------: | | [CoRal-project/roest-wav2vec2-315M-v2](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v2) | 315M | Read-aloud and conversation | 6.5% ± 0.2% | 16.3% ± 0.4% | | [CoRal-project/roest-whisper-large-v1](https://huggingface.co/CoRal-project/roest-whisper-large) | 1540M | Read-aloud | **4.3% ± 0.2%** | **10.4% ± 0.3%** | | [CoRal-project/roest-wav2vec2-315M-v1](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v1) | 315M | Read-aloud | 6.6% ± 0.2% | 17.0% ± 0.4% | | [mhenrichsen/hviske-v2](https://huggingface.co/syvai/hviske-v2) | 1540M | Read-aloud | 4.7% ± 0.2% | 11.8% ± 0.3% | | [openai/whisper-large-v3](https://hf.co/openai/whisper-large-v3) | 1540M | - | 11.4% ± 0.3% | 28.3% ± 0.6% | **OBS!** Benchmark for hviske-v2 has been reevaluted and the confidence interval is larger than reported in the model card. ### Detailed evaluation across demographics on the CoRal test data ### Table WER scores in % of evaluation across demographics on the CoRal test data | Category | roest-whisper-large-v1 | roest-wav2vec2-315m-v1 | roest-wav2vec2-315m-v2 | |:---:|:---:|:---:|:---:| | female | 11.5 | 18.5 | 17.7 | | male | 9.4 | 15.5 | 14.9 | | 0-25 | 9.0 | 14.7 | 14.0 | | 25-50 | 10.1 | 16.6 | 15.8 | | 50+ | 11.3 | 18.2 | 17.7 | | Bornholmsk | 9.8 | 17.7 | 15.7 | | Fynsk | 12.1 | 18.3 | 17.7 | | Københavnsk | 5.9 | 10.2 | 10.0 | | Non-native | 12.2 | 20.9 | 19.4 | | Nordjysk | 4.5 | 7.7 | 7.5 | | Sjællandsk | 7.6 | 12.6 | 12.7 | | Sydømål | 10.0 | 14.9 | 15.3 | | Sønderjysk | 17.5 | 26.0 | 25.4 | | Vestjysk | 15.0 | 26.3 | 25.2 | | Østjysk | 7.5 | 11.7 | 11.3 | | Overall | 10.4 | 17.0 | 16.3 | ### Table CER scores in % of evaluation across demographics on the CoRal test data | Category | roest-whisper-large-v1 | roest-wav2vec2-315m-v1 | roest-wav2vec2-315m-v2 | |:---:|:---:|:---:|:---:| | female | 5.1 | 7.4 | 7.2 | | male | 3.6 | 5.8 | 5.7 | | 0-25 | 3.4 | 5.4 | 5.3 | | 25-50 | 4.0 | 6.2 | 6.0 | | 50+ | 5.0 | 7.5 | 7.4 | | Bornholmsk | 3.8 | 6.8 | 6.1 | | Fynsk | 5.1 | 7.4 | 7.2 | | Københavnsk | 1.9 | 3.3 | 3.2 | | Non-native | 4.8 | 7.8 | 7.5 | | Nordjysk | 1.6 | 2.6 | 2.8 | | Sjællandsk | 3.0 | 4.4 | 4.5 | | Sydømål | 4.1 | 6.4 | 6.4 | | Sønderjysk | 8.8 | 11.9 | 11.6 | | Vestjysk | 6.4 | 10.1 | 9.8 | | Østjysk | 2.6 | 4.0 | 4.1 | | Overall | 4.3 | 6.6 | 6.5 | ### Roest-wav2vec2-315M with and without language model The inclusion of a post-processing language model can affect the performance significantly. The Roest-v1 and Roest-v2 models are using the same Language Model (LM). The utilized LM is the one trained and used by [alexandrainst/roest-wav2vec2-315m-v1](https://huggingface.co/alexandrainst/roest-315m). | Model | Number of parameters | Finetuned on data of type | Postprocessed with Language Model | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) CER | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) WER | | :-------------------------------------------------------------------------------------------- | -------------------: | --------------------------: | --------------------------------: | --------------------------------------------------------------------------------------: | --------------------------------------------------------------------------------------: | | [CoRal-project/roest-wav2vec2-315M-v2](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v2) | 315M | Read-aloud and conversation | Yes | **6.5% ± 0.2%** | **16.3% ± 0.4%** | | [CoRal-project/roest-wav2vec2-315M-v2](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v2) | 315M | Read-aloud and conversation | No | 8.2% ± 0.2% | 25.1% ± 0.4% | | [alexandrainst/roest-wav2vec2-315m-v1](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v1) | 315M | Read-aloud | Yes | 6.6% ± 0.2% | 17.0% ± 0.4% | | [alexandrainst/roest-wav2vec2-315m-v1](https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v1) | 315M | Read-aloud | No | 8.6% ± 0.2% | 26.3% ± 0.5% | ### Detailed Roest-wav2vec2-315M with and without language model on different dialects Here are the results of the model on different danish dialects in the test set: | | Roest-v1 | | Roest-v1 | | Roest-v2 | | Roest-v2 | | |-------------|---------|---------|---------|---------|---------|---------|---------|---------| | LM | No | | Yes | | No | | Yes | | |-------------|---------|---------|---------|---------|---------|---------|---------|---------| | Dialect | CER (%) | WER (%) | CER (%) | WER (%) | CER (%) | WER (%) | CER (%) | WER (%) | | Vestjysk | 12.7 | 37.1 | 10.1 | 26.3 | 12.2 | 36.3 | 9.82 | 25.2 | | Sønderjysk | 14.7 | 37.8 | 11.9 | 26.0 | 14.2 | 36.2 | 11.6 | 25.4 | | Bornholmsk | 9.32 | 29.9 | 6.79 | 17.7 | 8.08 | 26.7 | 6.12 | 15.7 | | Østjysk | 5.51 | 18.7 | 3.97 | 11.7 | 5.39 | 18.0 | 4.06 | 11.3 | | Nordjysk | 3.86 | 13.6 | 2.57 | 7.72 | 3.80 | 13.5 | 2.75 | 7.51 | | Københavnsk | 5.27 | 18.8 | 3.31 | 10.2 | 5.02 | 17.7 | 3.20 | 9.98 | | Fynsk | 9.41 | 28.6 | 7.43 | 18.3 | 8.86 | 27.0 | 7.20 | 17.7 | | Non-native | 10.6 | 33.2 | 7.84 | 20.9 | 10.0 | 31.6 | 7.46 | 19.4 | | Sjællandsk | 5.82 | 19.5 | 4.44 | 12.6 | 5.70 | 18.6 | 4.48 | 12.7 | | Sydømål | 7.09 | 20.7 | 6.38 | 14.9 | 6.96 | 20.4 | 6.44 | 15.3 | ### Performance on Other Datasets The model was also tested against other datasets to evaluate generalizability: | | **Roest-wav2vec2-315M-v1** | | **Roest-wav2vec2-315M-v2** | | | ------------------------------------------------------------------------------------- | ----------- | ----- | ----------- | -------- | | Evaluation Dataset | WER % | CER % | WER % | CER % | | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) | 17.0 | 6.6 | **16.3** | **6.5** | | [NST-da](https://huggingface.co/datasets/alexandrainst/nst-da) | 29.7 | 13.9 | **26.1** | **11.9** | | [CommonVoice17](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | 16.7 | 6.6 | **14.4** | **5.4** | | [Fleurs-da_dk](https://huggingface.co/datasets/google/fleurs) | 27.3 | 7.9 | **26.4** | **7.7** | | [Fleurs-da_dk](https://huggingface.co/datasets/google/fleurs) Normed | 16.6 | 6.3 | **15.6** | **6.1** | **OBS!** The vocab used for training incudes numerals (0,1,2,..,9), which are translated to text in a post-processing step. If the model misses spaces the numbers are interpreted as one, which expecially affects the NST score as this dataset contains many numerals. --- ## Training curves --- ## Creators and Funders This model has been trained and the model card written by Marie Juhl Jørgensen and Søren Vejlgaard Holm at [Alvenir](https://www.alvenir.ai/). The CoRal project is funded by the [Danish Innovation Fund](https://innovationsfonden.dk/) and consists of the following partners: - [Alexandra Institute](https://alexandra.dk/) - [University of Copenhagen](https://www.ku.dk/) - [Agency for Digital Government](https://digst.dk/) - [Alvenir](https://www.alvenir.ai/) - [Corti](https://www.corti.ai/) We would like specifically to thank Dan Saattrup Nielsen, Alexandra Institute for (among other things) the repository work and Simon Leminen Madsen, Alexandra Institute for modelling work. ## Citation We will submit a research paper soon, but until then, if you use this model in your research or development, please cite it as follows: @misc{roest-wav2vec2-315m-v2, author = {Marie Juhl Jørgensen, Søren Vejlgaard Holm, Martin Carsten Nielsen, Dan Saattrup Nielsen, Sif Bernstorff Lehmann, Simon Leminen Madsen, Anders Jess Pedersen, Anna Katrine van Zee, Anders Søgaard and Torben Blach}, title = {Roest-wav2vec-315m-v2: A Danish state-of-the-art speech recognition model trained on varied demographics and dialects}, year = {2025}, url = {https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v2}, }