transformers tensorboard safetensors whisper automatic-speech-recognition generated_from_trainer ga en dataset:ymoslem/IWSLT2023-GA-EN dataset:ymoslem/FLEURS-GA-EN dataset:ymoslem/BitesizeIrish-GA-EN dataset:ymoslem/SpokenWords-GA-EN-MTed dataset:ymoslem/Tatoeba-Speech-Irish dataset:ymoslem/Wikimedia-Speech-Irish dataset:ymoslem/EUbookshop-Speech-Irish base_model:openai/whisper-medium base_model:finetune:openai/whisper-medium license:apache-2.0 model-index endpoints_compatible region:us
---
base_model: openai/whisper-medium
datasets:
- ymoslem/IWSLT2023-GA-EN
- ymoslem/FLEURS-GA-EN
- ymoslem/BitesizeIrish-GA-EN
- ymoslem/SpokenWords-GA-EN-MTed
- ymoslem/Tatoeba-Speech-Irish
- ymoslem/Wikimedia-Speech-Irish
- ymoslem/EUbookshop-Speech-Irish
language:
- ga
- en
license: apache-2.0
metrics:
- bleu
- wer
tags:
- generated_from_trainer
model-index:
- name: Whisper Medium GA-EN Speech Translation
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      name: IWSLT-2023, FLEURS, BiteSize, SpokenWords, Tatoeba, Wikimedia, and EUbookshop
      type: ymoslem/IWSLT2023-GA-EN
    metrics:
    - type: bleu
      value: 34.85
      name: Bleu
    - type: wer
      value: 60.91850517784781
      name: Wer
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Whisper Medium GA-EN Speech Translation

This model is a fine-tuned version of [openai/whisper-medium](https://huggingface.co/openai/whisper-medium) on the IWSLT-2023, FLEURS, BiteSize, SpokenWords, Tatoeba, Wikimedia, and EUbookshop dataset.
It achieves the following results on the evaluation set:
- Loss: 1.2038
- Bleu: 34.85
- Chrf: 54.43
- Wer: 60.9185

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.03
- training_steps: 15000
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step  | Bleu  | Chrf  | Validation Loss | Wer      |
|:-------------:|:------:|:-----:|:-----:|:-----:|:---------------:|:--------:|
| 2.5219        | 0.0138 | 100   | 0.44  | 10.48 | 2.1106          | 107.2490 |
| 2.4608        | 0.0276 | 200   | 3.3   | 20.43 | 2.1816          | 179.1535 |
| 2.3008        | 0.0414 | 300   | 3.66  | 21.59 | 2.0587          | 206.4836 |
| 2.2095        | 0.0552 | 400   | 8.79  | 27.66 | 1.9459          | 100.3602 |
| 2.0454        | 0.0690 | 500   | 8.14  | 27.36 | 1.8681          | 122.1522 |
| 1.9937        | 0.0828 | 600   | 11.05 | 30.26 | 1.8717          | 97.2535  |
| 1.868         | 0.0966 | 700   | 9.14  | 29.03 | 1.7917          | 129.0410 |
| 1.9924        | 0.1103 | 800   | 12.62 | 33.2  | 1.7170          | 89.6443  |
| 1.8646        | 0.1241 | 900   | 11.98 | 30.77 | 1.7252          | 97.8838  |
| 1.7644        | 0.1379 | 1000  | 10.87 | 31.0  | 1.6832          | 109.1851 |
| 1.692         | 0.1517 | 1100  | 13.05 | 34.46 | 1.6837          | 93.3814  |
| 1.7044        | 0.1655 | 1200  | 20.95 | 37.42 | 1.5527          | 75.2364  |
| 1.6824        | 0.1793 | 1300  | 14.91 | 35.56 | 1.5611          | 92.6159  |
| 1.6557        | 0.1931 | 1400  | 14.0  | 36.54 | 1.5554          | 99.8199  |
| 1.5456        | 0.2069 | 1500  | 19.72 | 39.81 | 1.5058          | 83.5660  |
| 1.3755        | 0.2207 | 1600  | 18.04 | 37.95 | 1.5039          | 82.9806  |
| 1.3959        | 0.2345 | 1700  | 17.01 | 39.5  | 1.4374          | 85.2319  |
| 1.5012        | 0.2483 | 1800  | 14.93 | 39.24 | 1.4242          | 114.4079 |
| 1.4278        | 0.2621 | 1900  | 23.85 | 42.69 | 1.3904          | 73.0302  |
| 1.3285        | 0.2759 | 2000  | 17.7  | 37.23 | 1.4493          | 83.8811  |
| 1.2655        | 0.2897 | 2100  | 20.1  | 40.32 | 1.3661          | 79.7839  |
| 1.2074        | 0.3034 | 2200  | 24.45 | 43.79 | 1.3387          | 72.9851  |
| 1.1893        | 0.3172 | 2300  | 21.45 | 42.61 | 1.3308          | 82.3953  |
| 1.1236        | 0.3310 | 2400  | 22.77 | 44.17 | 1.3050          | 77.3075  |
| 1.0934        | 0.3448 | 2500  | 25.54 | 46.32 | 1.2793          | 72.2647  |
| 1.06          | 0.3586 | 2600  | 28.27 | 47.32 | 1.2396          | 65.6911  |
| 1.0327        | 0.3724 | 2700  | 28.45 | 47.01 | 1.2577          | 67.3570  |
| 1.1623        | 0.3862 | 2800  | 24.54 | 47.43 | 1.2194          | 73.6155  |
| 1.0215        | 0.4    | 2900  | 27.4  | 49.6  | 1.2039          | 69.2481  |
| 0.9185        | 0.4138 | 3000  | 27.04 | 49.24 | 1.1724          | 67.8973  |
| 0.9003        | 0.4276 | 3100  | 31.08 | 50.11 | 1.1674          | 63.8001  |
| 0.9839        | 0.4414 | 3200  | 30.24 | 50.63 | 1.1580          | 64.5655  |
| 0.9396        | 0.4552 | 3300  | 30.79 | 51.72 | 1.1202          | 64.9257  |
| 0.9051        | 0.4690 | 3400  | 30.34 | 53.08 | 1.1180          | 66.4566  |
| 0.8621        | 0.4828 | 3500  | 33.3  | 53.86 | 1.1042          | 60.7834  |
| 0.8236        | 0.4966 | 3600  | 32.77 | 53.21 | 1.1070          | 62.0441  |
| 0.829         | 0.5103 | 3700  | 32.49 | 54.21 | 1.0771          | 62.5844  |
| 0.8375        | 0.5241 | 3800  | 32.27 | 53.98 | 1.0780          | 63.0797  |
| 0.8206        | 0.5379 | 3900  | 33.26 | 55.07 | 1.0615          | 61.6389  |
| 0.8059        | 0.5517 | 4000  | 33.24 | 55.16 | 1.0552          | 61.5038  |
| 0.9133        | 0.5655 | 4100  | 29.38 | 49.22 | 1.2218          | 66.0964  |
| 1.051         | 0.5793 | 4200  | 25.12 | 46.01 | 1.2304          | 71.8145  |
| 0.954         | 0.5931 | 4300  | 25.47 | 45.88 | 1.2501          | 75.3715  |
| 0.939         | 0.6069 | 4400  | 29.19 | 47.63 | 1.2204          | 66.9068  |
| 0.9887        | 0.6207 | 4500  | 27.99 | 47.01 | 1.2099          | 67.7172  |
| 1.0044        | 0.6345 | 4600  | 23.77 | 45.33 | 1.2080          | 73.3904  |
| 0.9881        | 0.6483 | 4700  | 26.46 | 47.36 | 1.2188          | 68.5277  |
| 0.9674        | 0.6621 | 4800  | 26.11 | 45.92 | 1.2296          | 68.3026  |
| 0.8845        | 0.6759 | 4900  | 27.3  | 46.08 | 1.2347          | 68.0324  |
| 0.8297        | 0.6897 | 5000  | 29.48 | 48.96 | 1.2108          | 64.6105  |
| 0.9065        | 0.7034 | 5100  | 29.81 | 49.94 | 1.1873          | 64.2503  |
| 0.8096        | 0.7172 | 5200  | 28.5  | 46.93 | 1.2122          | 66.2314  |
| 0.8077        | 0.7310 | 5300  | 29.26 | 48.21 | 1.1945          | 64.4755  |
| 0.8227        | 0.7448 | 5400  | 26.82 | 48.43 | 1.2310          | 71.4093  |
| 0.7587        | 0.7586 | 5500  | 29.45 | 49.03 | 1.2067          | 65.3309  |
| 0.7206        | 0.7724 | 5600  | 29.89 | 49.33 | 1.2114          | 65.5561  |
| 0.8088        | 0.7862 | 5700  | 31.88 | 51.4  | 1.1689          | 64.2954  |
| 0.693         | 0.8    | 5800  | 27.23 | 48.11 | 1.1644          | 68.7078  |
| 0.7099        | 0.8138 | 5900  | 31.01 | 49.42 | 1.1852          | 63.3949  |
| 0.7564        | 0.8276 | 6000  | 28.3  | 50.34 | 1.1554          | 71.0941  |
| 0.584         | 0.8414 | 6100  | 34.79 | 51.69 | 1.1566          | 59.0725  |
| 0.6817        | 0.8552 | 6200  | 34.08 | 51.95 | 1.1245          | 59.8829  |
| 0.5968        | 0.8690 | 6300  | 32.4  | 51.59 | 1.1475          | 62.9896  |
| 0.6092        | 0.8828 | 6400  | 32.83 | 52.82 | 1.1250          | 62.5844  |
| 0.6325        | 0.8966 | 6500  | 29.29 | 51.68 | 1.1108          | 69.1130  |
| 0.6002        | 0.9103 | 6600  | 27.64 | 52.7  | 1.0993          | 71.0941  |
| 0.6247        | 0.9241 | 6700  | 28.39 | 52.4  | 1.0898          | 68.3026  |
| 0.6257        | 0.9379 | 6800  | 28.54 | 52.33 | 1.0863          | 70.9140  |
| 0.6719        | 0.9517 | 6900  | 31.43 | 53.53 | 1.0891          | 66.1414  |
| 0.4994        | 0.9655 | 7000  | 33.81 | 52.77 | 1.1066          | 61.0986  |
| 0.5469        | 0.9793 | 7100  | 30.52 | 53.13 | 1.0891          | 67.3570  |
| 0.6031        | 0.9931 | 7200  | 33.16 | 54.03 | 1.0933          | 62.1792  |
| 0.2469        | 1.0069 | 7300  | 33.76 | 52.38 | 1.1426          | 62.8546  |
| 0.2572        | 1.0207 | 7400  | 33.16 | 51.71 | 1.1292          | 64.8807  |
| 0.2762        | 1.0345 | 7500  | 34.76 | 54.28 | 1.1090          | 60.7384  |
| 0.2332        | 1.0483 | 7600  | 30.95 | 52.28 | 1.1073          | 66.1864  |
| 0.2069        | 1.0621 | 7700  | 32.39 | 53.08 | 1.0999          | 65.5561  |
| 0.2417        | 1.0759 | 7800  | 31.3  | 53.87 | 1.1008          | 65.1058  |
| 0.2403        | 1.0897 | 7900  | 32.18 | 53.3  | 1.1053          | 66.4566  |
| 0.208         | 1.1034 | 8000  | 32.0  | 52.48 | 1.1067          | 66.7717  |
| 0.3328        | 1.1172 | 8100  | 28.92 | 49.12 | 1.2137          | 68.4376  |
| 0.4045        | 1.1310 | 8200  | 28.47 | 51.53 | 1.2165          | 68.3926  |
| 0.4175        | 1.1448 | 8300  | 26.88 | 47.57 | 1.2790          | 74.5160  |
| 0.3976        | 1.1586 | 8400  | 21.56 | 44.64 | 1.3060          | 84.1513  |
| 0.4026        | 1.1724 | 8500  | 25.22 | 47.73 | 1.2476          | 73.1202  |
| 0.4088        | 1.1862 | 8600  | 26.03 | 48.08 | 1.2387          | 72.8050  |
| 0.4245        | 1.2    | 8700  | 29.8  | 49.69 | 1.2136          | 67.4021  |
| 0.4083        | 1.2138 | 8800  | 26.26 | 48.23 | 1.2784          | 73.4804  |
| 0.3832        | 1.2276 | 8900  | 29.06 | 49.36 | 1.2527          | 66.4115  |
| 0.4335        | 1.2414 | 9000  | 30.11 | 49.24 | 1.2772          | 67.2670  |
| 0.4056        | 1.2552 | 9100  | 32.51 | 50.18 | 1.3013          | 63.3048  |
| 0.3877        | 1.2690 | 9200  | 26.91 | 47.47 | 1.2897          | 71.5894  |
| 0.3787        | 1.2828 | 9300  | 1.2430| 30.16 | 50.61           | 65.1058  |
| 0.3947        | 1.2966 | 9400  | 1.2318| 29.9  | 50.77           | 66.0964  |
| 0.3908        | 1.3103 | 9500  | 1.1927| 30.7  | 51.62           | 64.6105  |
| 0.405         | 1.3241 | 9600  | 1.2249| 26.56 | 49.05           | 71.7695  |
| 0.3847        | 1.3379 | 9700  | 1.2105| 33.22 | 51.98           | 61.8640  |
| 0.3674        | 1.3517 | 9800  | 1.2545| 30.93 | 50.34           | 65.6011  |
| 0.3642        | 1.3655 | 9900  | 1.2443| 25.23 | 47.97           | 77.9379  |
| 0.3636        | 1.3793 | 10000 | 1.2796| 26.78 | 48.07           | 73.6155  |
| 0.329         | 1.3931 | 10100 | 1.2373| 29.06 | 49.55           | 66.4566  |
| 0.4195        | 1.4069 | 10200 | 1.2187| 29.11 | 50.65           | 66.2314  |
| 0.4244        | 1.4207 | 10300 | 1.2346| 27.97 | 49.86           | 69.0680  |
| 0.3338        | 1.4345 | 10400 | 1.2239| 29.96 | 50.45           | 66.0063  |
| 0.3401        | 1.4483 | 10500 | 1.2501| 29.84 | 51.0            | 65.6911  |
| 0.3792        | 1.4621 | 10600 | 1.2353| 28.38 | 49.19           | 69.1130  |
| 0.3549        | 1.4759 | 10700 | 1.2178| 28.63 | 49.73           | 68.5727  |
| 0.3326        | 1.4897 | 10800 | 1.1936| 29.57 | 51.1            | 64.4755  |
| 0.3418        | 1.5034 | 10900 | 1.1741| 33.06 | 52.86           | 60.9185  |
| 0.3143        | 1.5172 | 11000 | 1.2046| 31.49 | 50.4            | 63.5750  |
| 0.3245        | 1.5310 | 11100 | 1.2145| 30.9  | 50.17           | 64.6105  |
| 0.3268        | 1.5448 | 11200 | 1.2119| 33.5  | 53.0            | 60.2431  |
| 0.2894        | 1.5586 | 11300 | 1.2126| 32.01 | 52.17           | 61.0986  |
| 0.2702        | 1.5724 | 11400 | 1.2213| 31.33 | 50.89           | 63.7551  |
| 0.2876        | 1.5862 | 11500 | 1.2126| 31.44 | 51.28           | 63.1697  |
| 0.2759        | 1.6    | 11600 | 1.2283| 30.49 | 51.02           | 64.7456  |
| 0.2902        | 1.6138 | 11700 | 1.2205| 32.33 | 50.53           | 63.2148  |
| 0.2638        | 1.6276 | 11800 | 1.2097| 31.89 | 51.14           | 62.6745  |
| 0.2605        | 1.6414 | 11900 | 1.2129| 31.35 | 50.63           | 63.3048  |
| 0.2374        | 1.6552 | 12000 | 1.2319| 31.48 | 51.73           | 63.4849  |
| 0.2436        | 1.6690 | 12100 | 1.2219| 30.43 | 50.92           | 65.5110  |
| 0.2366        | 1.6828 | 12200 | 1.2367| 31.64 | 51.14           | 64.7006  |
| 0.218         | 1.6966 | 12300 | 1.2142| 30.8  | 51.63           | 64.1153  |
| 0.2313        | 1.7103 | 12400 | 1.1877| 30.8  | 50.63           | 64.5655  |
| 0.2307        | 1.7241 | 12500 | 1.1817| 32.22 | 51.41           | 63.3498  |
| 0.2638        | 1.7379 | 12600 | 1.1514| 33.74 | 52.11           | 60.6033  |
| 0.2211        | 1.7517 | 12700 | 1.1563| 30.71 | 52.07           | 64.5655  |
| 0.197         | 1.7655 | 12800 | 1.1941| 32.22 | 52.9            | 62.8546  |
| 0.2307        | 1.7793 | 12900 | 1.1771| 32.83 | 52.96           | 62.7645  |
| 0.198         | 1.7931 | 13000 | 1.1908| 32.16 | 51.85           | 63.9352  |
| 0.1716        | 1.8069 | 13100 | 1.2065| 31.91 | 51.37           | 62.6294  |
| 0.2031        | 1.8207 | 13200 | 1.1745| 31.83 | 51.86           | 64.0252  |
| 0.1785        | 1.8345 | 13300 | 1.1607| 31.33 | 52.57           | 64.7006  |
| 0.2013        | 1.8483 | 13400 | 1.1785| 33.29 | 53.34           | 62.6745  |
| 0.1842        | 1.8621 | 13500 | 1.1723| 34.41 | 54.31           | 60.0630  |
| 0.2015        | 1.8759 | 13600 | 1.1859| 32.88 | 53.07           | 62.2692  |
| 0.1848        | 1.8897 | 13700 | 1.1668| 33.62 | 53.75           | 62.8095  |
| 0.1394        | 1.9034 | 13800 | 1.1734| 34.33 | 54.03           | 61.2787  |
| 0.1774        | 1.9172 | 13900 | 1.1735| 32.63 | 53.37           | 62.8996  |
| 0.1506        | 1.9310 | 14000 | 1.1768| 35.17 | 54.34           | 59.4327  |
| 0.1399        | 1.9448 | 14100 | 1.1827| 33.68 | 53.8            | 62.1792  |
| 0.1434        | 1.9586 | 14200 | 1.1721| 34.62 | 54.24           | 60.9185  |
| 0.1203        | 1.9724 | 14300 | 1.1733| 34.08 | 53.75           | 61.8190  |
| 0.1417        | 1.9862 | 14400 | 1.1615| 33.98 | 54.19           | 62.1792  |
| 0.1458        | 2.0    | 14500 | 1.1739| 33.65 | 53.31           | 62.9896  |
| 0.07          | 2.0138 | 14600 | 1.1916| 33.98 | 53.96           | 61.9090  |
| 0.051         | 2.0276 | 14700 | 1.1967| 34.13 | 54.36           | 61.1887  |
| 0.0481        | 2.0414 | 14800 | 1.2024| 34.06 | 54.38           | 61.4588  |
| 0.0574        | 2.0552 | 14900 | 1.2038| 34.23 | 54.08           | 61.2787  |
| 0.0621        | 2.0690 | 15000 | 1.2038| 34.85 | 54.43           | 60.9185  |


### Framework versions

- Transformers 4.41.2
- Pytorch 2.2.0+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1


## Citation

```
@inproceedings{moslem-2024-leveraging,
    title = "Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation",
    author = "Moslem, Yasmin",
    booktitle = "Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand (in-person and online)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.iwslt-1.31/",
    doi = "10.18653/v1/2024.iwslt-1.31",
    pages = "265--273",
    abstract = "This paper describes our system submission to the International Conference on Spoken Language Translation (IWSLT 2024) for Irish-to-English speech translation. We built end-to-end systems based on Whisper, and employed a number of data augmentation techniques, such as speech back-translation and noise augmentation. We investigate the effect of using synthetic audio data and discuss several methods for enriching signal diversity."
}
```
