Create README.md

README.md

@@ -0,0 +1,159 @@
+---
+license: mit
+language:
+- en
+base_model:
+- distil-whisper/distil-large-v3.5
+pipeline_tag: automatic-speech-recognition
+library_name: transformers
+model-index:
+- name: whisper-large-v3-singlish-DRAFT
+  results:
+  - task:
+      type: automatic-speech-recognition
+    dataset:
+      name: SASRBench-v1
+      type: mjwong/SASRBench-v1
+      split: test
+    metrics:
+      - name: WER
+        type: WER
+        value: 14.89
+tags:
+- whisper
+---
+
+# Whisper large-v3-singlish-DRAFT
+
+**Whisper large-v3-singlish-DRAFT** is a fine-tuned automatic speech recognition (ASR) model tailored specifically for Singlish. Based on OpenAI’s Whisper architecture, this model has been optimized with Singlish-centric data to better capture the distinctive phonetic patterns and vocabulary commonly found in Singlish speech. It is designed to serve as a lightweight draft model in speculative decoding pipelines, working in tandem with [Whisper large-v3-singlish](https://huggingface.co/mjwong/whisper-large-v3-singlish) as the target model to improve transcription speed while maintaining accuracy.
+
+## Model Details
+
+- **Developed by:** Ming Jie Wong
+- **Base Model:** [distil-whisper/distil-large-v3.5](https://huggingface.co/distil-whisper/distil-large-v3.5)
+- **Model Type:** Encoder-decoder
+- **Metrics:** Word Error Rate (WER)
+- **Languages Supported:** English (with a focus on Singlish)
+- **License:** MIT
+
+### Description
+
+Whisper-large-v3-singlish-DRAFT was trained using pseudo-labels generated by its target model, Whisper-large-v3-singlish. The target model transcribed 66.9k audio recordings sourced from Part 3 of the Same Room Environment Close-talk Microphone section of [IMDA’s National Speech Corpus (NSC)](https://www.imda.gov.sg/how-we-can-help/national-speech-corpus). This self-distillation approach ensures high alignment between the draft and target models, enabling effective speculative decoding in Singlish speech.
+
+The original Part 3 of the National Speech Corpus comprises approximately 1,000 hours of conversational speech from around 1,000 local English speakers, recorded in pairs. These conversations cover everyday topics and include interactive game-based dialogues. Recordings were conducted in two environments:
+- Same Room, where speakers shared a room and were recorded using a close-talk mic and a boundary mic.
+- Separate Room, where each speaker was recorded individually using a standing mic and a telephone (IVR).
+
+Audio segments for the internal dataset were extracted using these criteria:
+- **Minimum Word Count:** 10 words
+  
+  _This threshold was chosen to ensure that each audio segment contains sufficient linguistic context for the model to better understand instructions in Singlish. Shorter segments may bias the model towards specific utterances or phrases, limiting its overall comprehension._
+- **Maximum Duration:** 20 seconds
+
+  _This threshold was chosen to provide enough context for accurate transcription while minimizing noise and computational complexity for longer audio segments._
+- **Sampling Rate**: All audio segments are down-sampled to 16kHz.
+
+Full experiments details will be added soon.
+
+### Fine-Tuning Details
+
+We applied fine-tuning on a single A100-80GB GPU.
+
+#### Training Hyperparameters
+The following hyperparameters are used:
+- **batch_size**: 16
+- **gradient_accumulation_steps**: 1
+- **learning_rate**: 1e-6
+- **warmup_steps**: 300
+- **max_steps**: 5000
+- **fp16**: true
+- **eval_batch_size**: 16
+- **eval_step**: 300
+- **max_grad_norm**: 1.0
+- **generation_max_length**: 225
+
+### Benchmark Performance
+
+We evaluated the speculative decoding setup for Whisper large-v3-singlish on [SASRBench-v1](https://huggingface.co/datasets/mjwong/SASRBench-v1), a benchmark dataset for evaluating ASR performance on Singlish:
+
+#### Model Performance
+
+| **Model**                                                                                          | **Rel. RTFx** | **WER**   |
+|----------------------------------------------------------------------------------------------------|---------------|-----------|
+| [Whisper-large-v3-singlish](https://huggingface.co/mjwong/whisper-large-v3-singlish)               | 1.00          | 16.45%    |
+| [Whisper-large-v3-turbo-singlish](https://huggingface.co/mjwong/whisper-large-v3-turbo-singlish)   | 2.36          | 13.35%    |
+| Whisper-large-v3-singlish + [DRAFT](https://huggingface.co/mjwong/whisper-large-v3-singlish-DRAFT) | 2.20          | 14.89%    |
+
+
+#### Speculative Acceptance Rates
+
+| **Speculative Setup**                                                                                 | **Micro Avg Acceptance**  | **Macro Avg Acceptance** |
+|-------------------------------------------------------------------------------------------------------|---------------------------|--------------------------|
+| Whisper-large-v3-singlish + [DRAFT](https://huggingface.co/mjwong/whisper-large-v3-singlish-DRAFT)    | 38.00%                    | 42.00%                   |
+
+## Disclaimer
+
+While this model has been fine-tuned to better recognize Singlish, users may experience inaccuracies, biases, or unexpected outputs, particularly in challenging audio conditions or with speakers using non-standard variations. Use of this model is at your own risk; the developers and distributors are not liable for any consequences arising from its use. Please validate results before deploying in any sensitive or production environment.
+
+## How to use the model
+
+Whisper-large-v3-singlish-DRAFT can be leveraged as an assistant model in a speculative decoding setup with Whisper-large-v3-singlish as the target. The assistant model proposes initial tokens, which are selectively verified by the target model to accelerate inference without sacrificing accuracy.
+
+```python
+import torch
+from transformers import (
+    pipeline,
+    AutoModelForCausalLM,
+    AutoModelForSpeechSeq2Seq,
+    AutoProcessor
+)
+
+TARGET_REPO_NAME = "mjwong/whisper-large-v3-singlish"
+DRAFT_REPO_NAME = "mjwong/whisper-large-v3-singlish-DRAFT"
+
+# Select appropriate device and precision
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
+
+# Load the draft model (used as the assistant in speculative decoding)
+assistant_model = AutoModelForCausalLM.from_pretrained(
+    DRAFT_REPO_NAME, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
+)
+assistant_model.to(device)
+
+# Load the main target model (the high-accuracy decoder)
+model = AutoModelForSpeechSeq2Seq.from_pretrained(
+    TARGET_REPO_NAME, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
+)
+model.to(device)
+
+# Load processor (tokenizer + feature extractor)
+processor = AutoProcessor.from_pretrained(TARGET_REPO_NAME)
+
+# Create the ASR pipeline with speculative decoding
+pipe = pipeline(
+    "automatic-speech-recognition",
+    model=model,
+    tokenizer=processor.tokenizer,
+    feature_extractor=processor.feature_extractor,
+    max_new_tokens=128,
+    generate_kwargs={"assistant_model": assistant_model},
+    torch_dtype=torch_dtype,
+    device=device,
+)
+```
+
+You can then use this pipeline to transcribe audios of arbitrary length.
+
+```python
+from datasets import load_dataset
+dataset = load_dataset("mjwong/SASRBench-v1", split="test")
+sample = dataset[0]["audio"]
+
+result = pipe(sample)
+print(result["text"])
+```
+
+## Contact
+
+For more information, please reach out to [email protected].