Hugging Face's logo

mjwong
/
whisper-large-v3-singlish-DRAFT

Automatic Speech Recognition
Transformers
Safetensors
English
whisper
Eval Results
Model card Files Community
mjwong commited on
Commit
696cc8c
·
verified ·
1 Parent(s): fb24a22

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -12
README.md CHANGED
@@ -72,24 +72,33 @@ The following hyperparameters are used:
72
  - **max_grad_norm**: 1.0
73
  - **generation_max_length**: 225
74
 
75
- ### Benchmark Performance
76
 
77
- We evaluated the speculative decoding setup for Whisper large-v3-singlish on [SASRBench-v1](https://huggingface.co/datasets/mjwong/SASRBench-v1), a benchmark dataset for evaluating ASR performance on Singlish:
78
 
79
- #### Model Performance
80
 
81
- | **Model** | **Rel. RTFx** | **WER** |
82
- |----------------------------------------------------------------------------------------------------|---------------|-----------|
83
- | [Whisper-large-v3-singlish](https://huggingface.co/mjwong/whisper-large-v3-singlish) | 1.00 | 16.41% |
84
- | [Whisper-large-v3-turbo-singlish](https://huggingface.co/mjwong/whisper-large-v3-turbo-singlish) | 2.36 | 13.35% |
85
- | Whisper-large-v3-singlish + [DRAFT](https://huggingface.co/mjwong/whisper-large-v3-singlish-DRAFT) | 2.20 | 14.84% |
86
 
 
87
 
88
- #### Speculative Acceptance Rates
89
 
90
- | **Speculative Setup** | **Micro Avg Acceptance** | **Macro Avg Acceptance** |
91
- |-------------------------------------------------------------------------------------------------------|---------------------------|--------------------------|
92
- | Whisper-large-v3-singlish + [DRAFT](https://huggingface.co/mjwong/whisper-large-v3-singlish-DRAFT) | 38.00% | 42.00% |
 
 
 
 
 
 
 
 
 
 
 
 
 
93
 
94
  ## Disclaimer
95
 
 
72
  - **max_grad_norm**: 1.0
73
  - **generation_max_length**: 225
74
 
75
+ ## Benchmark Performance
76
 
 
77
 
78
+ We evaluated the speculative decoding setup for Whisper-large-v3-singlish on the following datasets:
79
 
80
+ - [SASRBench-v1](https://huggingface.co/datasets/mjwong/SASRBench-v1): A benchmark dataset for evaluating ASR performance on Singlish.
 
 
 
 
81
 
82
+ - [AMI](https://huggingface.co/datasets/edinburghcstr/ami): A widely used dataset for meeting transcription and diarization tasks.
83
 
84
+ ### Model Performance
85
 
86
+ | **Dataset** | **Model Variant** | **Link** | **Rel. RTFx** | **WER** |
87
+ |-----------------|---------------------------|--------------------------------------------------------------------------------------------------|---------------|------------|
88
+ | SASRBench-v1 | Large | [Whisper-large-v3-singlish](https://huggingface.co/mjwong/whisper-large-v3-singlish) | 1.00 | 16.41% |
89
+ | SASRBench-v1 | Large-Turbo | [Whisper-large-v3-turbo-singlish](https://huggingface.co/mjwong/whisper-large-v3-turbo-singlish) | **2.36** | **13.35%** |
90
+ | SASRBench-v1 | Draft-enhanced Large | Whisper-large-v3-singlish + [DRAFT](https://huggingface.co/mjwong/whisper-large-v3-singlish-DRAFT) | 2.20 | 14.84% |
91
+ ||||||
92
+ | AMI | Large | [Whisper-large-v3-singlish](https://huggingface.co/mjwong/whisper-large-v3-singlish) | 1.00 | 23.72% |
93
+ | AMI | Large-Turbo | [Whisper-large-v3-turbo-singlish](https://huggingface.co/mjwong/whisper-large-v3-turbo-singlish) | 1.53 | **16.99%** |
94
+ | AMI | Draft-enhanced Large | Whisper-large-v3-singlish + [DRAFT](https://huggingface.co/mjwong/whisper-large-v3-singlish-DRAFT) | **2.27** | 22.06% |
95
+
96
+ ### Speculative Acceptance Rates (DRAFT-enhanced Large Model)
97
+
98
+ | **Dataset** | **Micro Avg Acceptance** | **Macro Avg Acceptance** |
99
+ |----------------|--------------------------|---------------------------|
100
+ | SASRBench-v1 | 38.00% | 42.00% |
101
+ | AMI | 38.00% | 43.00% |
102
 
103
  ## Disclaimer
104