IPA-Transcription-EN

Running

App Files Files Community

arunasrivastava commited on Dec 21, 2024

Commit

3a023fb

1 Parent(s): 03346c0

without eval library

Browse files

Files changed (12) hide show

__pycache__/main.cpython-310.pyc +0 -0
__pycache__/phone_metrics.cpython-310.pyc +0 -0
app.py +3 -2
constants.py +0 -108
img/logo-white.png +0 -0
install.sh +11 -0
main.py +2 -1
phone_metrics.py +106 -0
queue/leaderboard.json +80 -0
queue/results.json +460 -0
queue/tasks.json +129 -0
utils_display.py +0 -31

__pycache__/main.cpython-310.pyc CHANGED Viewed

Binary files a/__pycache__/main.cpython-310.pyc and b/__pycache__/main.cpython-310.pyc differ

__pycache__/phone_metrics.cpython-310.pyc ADDED Viewed

Binary file (4.16 kB). View file

app.py CHANGED Viewed

@@ -147,7 +147,8 @@ with gr.Blocks(css="""
         border-bottom: 1px solid #dddddd;
     }
 """) as demo:
-    gr.Markdown("# 🎯 Phonemic Transcription Model Evaluation Leaderboard")
     gr.Markdown("""
     ## Explanation of Metrics
     - **PER (Phoneme Error Rate)**: The Levenshtein distance calculated between phoneme sequences of the predicted and actual transcriptions.
@@ -158,7 +159,7 @@ with gr.Blocks(css="""
     The test set used for evaluation is from the [TIMIT speech corpus](https://www.kaggle.com/datasets/mfekadu/darpa-timit-acousticphonetic-continuous-speech). The TIMIT corpus is a widely used dataset for speech recognition research.
     ## Compute
-    This leaderboard uses the free basic plan (16GB RAM, 2vCPUs). The evaluation may take several hours to complete. Please be patient and do not submit the same model multiple times.
     """)
     with gr.Tabs():
         with gr.TabItem("🏆 Leaderboard"):

         border-bottom: 1px solid #dddddd;
     }
 """) as demo:
+    gr.Markdown("# 🎯 Phonemic Transcription Leaderboard")
+    gr.Markdown("#### Developed By: Koel Labs")
     gr.Markdown("""
     ## Explanation of Metrics
     - **PER (Phoneme Error Rate)**: The Levenshtein distance calculated between phoneme sequences of the predicted and actual transcriptions.
     The test set used for evaluation is from the [TIMIT speech corpus](https://www.kaggle.com/datasets/mfekadu/darpa-timit-acousticphonetic-continuous-speech). The TIMIT corpus is a widely used dataset for speech recognition research.
     ## Compute
+    This leaderboard uses the free basic plan (16GB RAM, 2vCPUs) to allow for reproducability. The evaluation may take several hours to complete. Please be patient and do not submit the same model multiple times.
     """)
     with gr.Tabs():
         with gr.TabItem("🏆 Leaderboard"):

constants.py DELETED Viewed

@@ -1,108 +0,0 @@
-from pathlib import Path
-# Directory where request by models are stored
-DIR_OUTPUT_REQUESTS = Path("requested_models")
-EVAL_REQUESTS_PATH = Path("eval_requests")
-##########################
-# Text definitions       #
-##########################
-banner_url = "https://huggingface.co/datasets/reach-vb/random-images/resolve/main/asr_leaderboard.png"
-BANNER = f'<div style="display: flex; justify-content: space-around;"><img src="{banner_url}" alt="Banner" style="width: 40vw; min-width: 300px; max-width: 600px;"> </div>'
-TITLE = "<html> <head> <style> h1 {text-align: center;} </style> </head> <body> <h1> 🤗 IPA Transcription Leaderboard </b> </body> </html>"
-INTRODUCTION_TEXT = "📐 The 🤗 IPA transcription Leaderboard ranks and evaluates speech recognition models \
-    on the Hugging Face Hub. \
-    \nWe report the Average [PER](https://huggingface.co/spaces/evaluate-metric/wer) (⬇️ lower the better) and [RTFx](https://github.com/NVIDIA/DeepLearningExamples/blob/master/Kaldi/SpeechRecognition/README.md#metrics) (⬆️ higher the better). Models are ranked based on their Average WER, from lowest to highest. Check the 📈 Metrics tab to understand how the models are evaluated. \
-    \nIf you want results for a model that is not listed here, you can submit a request for it to be included ✉️✨. \
-    \nThe leaderboard currently focuses on English speech recognition, and will be expanded to multilingual evaluation in later versions."
-METRICS_TAB_TEXT = """
-Here you will find details about the speech recognition metrics and datasets reported in our leaderboard.
-## Metrics
-Models are evaluated jointly using the Word Error Rate (WER) and Inverse Real Time Factor (RTFx) metrics. The WER metric
-is used to assess the accuracy of a system, and the RTFx the inference speed. Models are ranked in the leaderboard based
-on their WER, lowest to highest.
-Crucially, the WER and RTFx values are computed for the same inference run using a single script. The implication of this is two-fold:
-1. The WER and RTFx values are coupled: for a given WER, one can expect to achieve the corresponding RTFx. This allows the proposer to trade-off lower WER for higher RTFx should they wish.
-2. The WER and RTFx values are averaged over all audios in the benchmark (in the order of thousands of audios).
-For details on reproducing the benchmark numbers, refer to the [Open ASR GitHub repository](https://github.com/huggingface/open_asr_leaderboard#evaluate-a-model).
-### Word Error Rate (WER)
-Word Error Rate is used to measure the **accuracy** of automatic speech recognition systems. It calculates the percentage
-of words in the system's output that differ from the reference (correct) transcript. **A lower WER value indicates higher accuracy**.
-Take the following example:
-| Reference:  | the | cat | sat     | on  | the | mat |
-|-------------|-----|-----|---------|-----|-----|-----|
-| Prediction: | the | cat | **sit** | on  | the |     |  |
-| Label:      | ✅   | ✅   | S       | ✅   | ✅   | D   |
-Here, we have:
-* 1 substitution ("sit" instead of "sat")
-* 0 insertions
-* 1 deletion ("mat" is missing)
-This gives 2 errors in total. To get our word error rate, we divide the total number of errors (substitutions + insertions + deletions) by the total number of words in our
-reference (N), which for this example is 6:
-```
-WER = (S + I + D) / N = (1 + 0 + 1) / 6 = 0.333
-```
-Giving a WER of 0.33, or 33%. For a fair comparison, we calculate **zero-shot** (i.e. pre-trained models only) *normalised WER* for all the model checkpoints, meaning punctuation and casing is removed from the references and predictions. You can find the evaluation code on our [Github repository](https://github.com/huggingface/open_asr_leaderboard). To read more about how the WER is computed, refer to the [Audio Transformers Course](https://huggingface.co/learn/audio-course/chapter5/evaluation).
-### Inverse Real Time Factor (RTFx)
-Inverse Real Time Factor is a measure of  the **latency** of automatic speech recognition systems, i.e. how long it takes an
-model to process a given amount of speech. It is defined as:
-```
-RTFx = (number of seconds of audio inferred) / (compute time in seconds)
-```
-Therefore, and RTFx of 1 means a system processes speech as fast as it's spoken, while an RTFx of 2 means it takes half the time.
-Thus, **a higher RTFx value indicates lower latency**.
-## How to reproduce our results
-The ASR Leaderboard will be a continued effort to benchmark open source/access speech recognition models where possible.
-Along with the Leaderboard we're open-sourcing the codebase used for running these evaluations.
-For more details head over to our repo at: https://github.com/huggingface/open_asr_leaderboard
-P.S. We'd love to know which other models you'd like us to benchmark next. Contributions are more than welcome! ♥️
-## Benchmark datasets
-Evaluating Speech Recognition systems is a hard problem. We use the multi-dataset benchmarking strategy proposed in the
-[ESB paper](https://arxiv.org/abs/2210.13352) to obtain robust evaluation scores for each model.
-ESB is a benchmark for evaluating the performance of a single automatic speech recognition (ASR) system across a broad
-set of speech datasets. It comprises eight English speech recognition datasets, capturing a broad range of domains,
-acoustic conditions, speaker styles, and transcription requirements. As such, it gives a better indication of how
-a model is likely to perform on downstream ASR compared to evaluating it on one dataset alone.
-The ESB score is calculated as a macro-average of the WER scores across the ESB datasets. The models in the leaderboard
-are ranked based on their average WER scores, from lowest to highest.
-We are currently working to add and curate more datasets. Right now, models will be evaluated just on the TIMIT test dataset for phoneme transcription.
-| Dataset                                                                                 | Domain                      | Speaking Style        | Train (h) | Dev (h) | Test (h) | Transcriptions     | License         |
-|-----------------------------------------------------------------------------------------|-----------------------------|-----------------------|-----------|---------|----------|--------------------|-----------------|
-| [TIMIT Dataset](https://www.kaggle.com/datasets/mfekadu/darpa-timit-acousticphonetic-continuous-speech)                          | Audiobook                   | Narrated              | 960       | 11      | 11       | Normalised         | CC-BY-4.0       |
-For more details on the individual datasets and how models are evaluated to give the ESB score, refer to the [ESB paper](https://arxiv.org/abs/2210.13352).
-"""
-LEADERBOARD_CSS = """
-#leaderboard-table th .header-content {
-    white-space: nowrap;
-}
-"""

img/logo-white.png ADDED Viewed

install.sh ADDED Viewed

	@@ -0,0 +1,11 @@

+# Create a virtual environment with Python 3.10
+python3.10 -m venv venv
+# Activate the virtual environment
+source venv/bin/activate
+# Install the required dependencies
+pip install -r requirements.txt
+# Run the application
+uvicorn app:app --host 0.0.0.0 --port 7860

main.py CHANGED Viewed

@@ -14,6 +14,7 @@ import os
 from pathlib import Path
 from huggingface_hub import HfApi
 import evaluate
 # Set up download configuration with your token
@@ -42,7 +43,7 @@ PATHS = {
 }
 # Initialize evaluation metric
-phone_errors = evaluate.load("ginic/phone_errors")
 class TimitDataManager:

 from pathlib import Path
 from huggingface_hub import HfApi
 import evaluate
+from phone_metrics import PhoneErrorMetrics
 # Set up download configuration with your token
 }
 # Initialize evaluation metric
+phone_errors = PhoneErrorMetrics()
 class TimitDataManager:

phone_metrics.py ADDED Viewed

	@@ -0,0 +1,106 @@

+# phone_metrics.py
+"""
+This module implements phone error metrics based on the work from ginic/phone_errors.
+Original implementation: https://huggingface.co/spaces/ginic/phone_errors
+Citation:
+@inproceedings{Mortensen-et-al:2016,
+  author    = {David R. Mortensen and
+               Patrick Littell and
+               Akash Bharadwaj and
+               Kartik Goyal and
+               Chris Dyer and
+               Lori S. Levin},
+  title     = {PanPhon: {A} Resource for Mapping {IPA} Segments to Articulatory Feature Vectors},
+  booktitle = {Proceedings of {COLING} 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
+  pages     = {3475--3484},
+  publisher = {{ACL}},
+  year      = {2016}
+}
+"""
+import numpy as np
+import panphon.distance
+from typing import List, Dict
+class PhoneErrorMetrics:
+    def __init__(self, feature_model: str = "segment"):
+        """Initialize the phone error metrics calculator.
+        Args:
+            feature_model (str): panphon feature parsing model ("strict", "permissive", or "segment")
+        """
+        self.distance_computer = panphon.distance.Distance(feature_model=feature_model)
+    def _phone_error_rate(self, prediction: str, reference: str) -> float:
+        """Compute phone error rate between prediction and reference.
+        Args:
+            prediction (str): Predicted IPA string
+            reference (str): Reference IPA string
+        Returns:
+            float: Phone error rate
+        """
+        if not reference:
+            raise ValueError("Reference string cannot be empty")
+        pred_phones = self.distance_computer.fm.ipa_segs(prediction)
+        ref_phones = self.distance_computer.fm.ipa_segs(reference)
+        phone_edits = self.distance_computer.min_edit_distance(
+            lambda x: 1,  # deletion cost
+            lambda x: 1,  # insertion cost
+            lambda x, y: 0 if x == y else 1,  # substitution cost
+            [[]],
+            pred_phones,
+            ref_phones
+        )
+        return phone_edits / len(ref_phones)
+    def compute(self,
+                predictions: List[str],
+                references: List[str],
+                is_normalize_pfer: bool = False) -> Dict:
+        """Compute phone error metrics between predictions and references.
+        Args:
+            predictions (List[str]): List of predicted IPA strings
+            references (List[str]): List of reference IPA strings
+            is_normalize_pfer (bool): Whether to normalize phone feature error rates
+        Returns:
+            Dict containing:
+                - phone_error_rates: List of PER for each pair
+                - mean_phone_error_rate: Average PER
+                - phone_feature_error_rates: List of PFER for each pair
+                - mean_phone_feature_error_rate: Average PFER
+                - feature_error_rates: List of FER for each pair
+                - mean_feature_error_rate: Average FER
+        """
+        phone_error_rates = []
+        feature_error_rates = []
+        hamming_distances = []
+        for pred, ref in zip(predictions, references):
+            if is_normalize_pfer:
+                hd = self.distance_computer.hamming_feature_edit_distance_div_maxlen(pred, ref)
+            else:
+                hd = self.distance_computer.hamming_feature_edit_distance(pred, ref)
+            hamming_distances.append(hd)
+            per = self._phone_error_rate(pred, ref)
+            phone_error_rates.append(per)
+            fer = self.distance_computer.feature_error_rate(pred, ref)
+            feature_error_rates.append(fer)
+        return {
+            "phone_error_rates": phone_error_rates,
+            "mean_phone_error_rate": float(np.mean(phone_error_rates)),
+            "phone_feature_error_rates": hamming_distances,
+            "mean_phone_feature_error_rate": float(np.mean(hamming_distances)),
+            "feature_error_rates": feature_error_rates,
+            "mean_feature_error_rate": float(np.mean(feature_error_rates))
+        }

queue/leaderboard.json CHANGED Viewed

@@ -98,5 +98,85 @@
         "subset": "test",
         "github_url": "https://huggingface.co/ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
         "submission_date": "2024-12-18T23:29:27.322286"
     }
 ]

         "subset": "test",
         "github_url": "https://huggingface.co/ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
         "submission_date": "2024-12-18T23:29:27.322286"
+    },
+    {
+        "submission_id": "d0b2f8b4-20f8-45b4-b1a5-c81390d75b29",
+        "submission_name": "wav2vec2 non-english transcription",
+        "model": "ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
+        "average_per": 0.6417205190285036,
+        "average_pwed": 0.19048963968896404,
+        "subset": "test",
+        "github_url": "https://huggingface.co/ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
+        "submission_date": "2024-12-19T07:41:18.135985"
+    },
+    {
+        "submission_id": "3bbb0f03-31a5-45b0-bde3-bbf574f19983",
+        "submission_name": "phonetic transcription with the Buckeye corpus, from xlsr-53 model",
+        "model": "ginic/gender_split_70_female_4_wav2vec2-large-xlsr-buckeye-ipa",
+        "average_per": 0.2810165988557621,
+        "average_pwed": 0.10703377161801164,
+        "subset": "test",
+        "github_url": "https://github.com/ginic/multipa/tree/buckeye_experiments",
+        "submission_date": "2024-12-20T13:45:52.010575"
+    },
+    {
+        "submission_id": "2ed095f7-4712-4539-87b6-1e8588ac92a3",
+        "submission_name": "phonetic transcription",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-INTERNATIONAL1.9.2WithoutSpaces",
+        "average_per": 0.9537775908999574,
+        "average_pwed": 0.9351204819224959,
+        "subset": "test",
+        "github_url": "https://huggingface.co/Jubliano/wav2vec2-large-xls-r-300m-ipa-INTERNATIONAL1.5WithoutSpaces",
+        "submission_date": "2024-12-20T14:21:32.293694"
+    },
+    {
+        "submission_id": "9cf02ce8-fc43-4d23-a8bb-b44e3116a93c",
+        "submission_name": "Jubliano xlsr model",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-nl",
+        "average_per": 0.9887075544197294,
+        "average_pwed": 0.9692486915717254,
+        "subset": "test",
+        "github_url": "https://huggingface.co/Jubliano/wav2vec2-large-xls-r-300m-ipa-nl1.1",
+        "submission_date": "2024-12-20T15:40:51.632895"
+    },
+    {
+        "submission_id": "d5013845-f5c9-428a-8b39-7db066bb9f05",
+        "submission_name": "speech31 phoneme transcription english",
+        "model": "speech31/wavlm-large-english-ipa",
+        "average_per": 0.3694017596969614,
+        "average_pwed": 0.1356824900612308,
+        "subset": "test",
+        "github_url": "https://huggingface.co/speech31/wavlm-large-english-ipa",
+        "submission_date": "2024-12-20T16:26:47.982209"
+    },
+    {
+        "submission_id": "362c788d-bc2e-427d-8c74-105f6235cf62",
+        "submission_name": "speech31 xlsr model",
+        "model": "speech31/XLS-R-300m-english-ipa",
+        "average_per": 0.36382554692045954,
+        "average_pwed": 0.1299702312124616,
+        "subset": "test",
+        "github_url": "https://huggingface.co/speech31/XLS-R-300m-english-ipa",
+        "submission_date": "2024-12-20T16:47:54.826509"
+    },
+    {
+        "submission_id": "49e22782-0af1-4313-bc0c-60cb2f28d78f",
+        "submission_name": "model is a fine-tuned version of facebook/wav2vec2-large on the TIMIT dataset",
+        "model": "speech31/wav2vec2-large-english-TIMIT-phoneme_v3",
+        "average_per": 0.44563344149564776,
+        "average_pwed": 0.18844914029048124,
+        "subset": "test",
+        "github_url": "https://huggingface.co/speech31/wav2vec2-large-english-TIMIT-phoneme_v3",
+        "submission_date": "2024-12-20T17:05:35.213738"
+    },
+    {
+        "submission_id": "26c04108-1131-435c-95f1-bb56b2aff06c",
+        "submission_name": "fine-tuned version of facebook/wav2vec2-large on the None dataset",
+        "model": "speech31/wav2vec2-large-TIMIT-IPA2",
+        "average_per": 0.4847029843149011,
+        "average_pwed": 0.2072006544586948,
+        "subset": "test",
+        "github_url": "https://huggingface.co/speech31/wav2vec2-large-TIMIT-IPA2",
+        "submission_date": "2024-12-20T22:50:50.645178"
     }
 ]

queue/results.json CHANGED Viewed

@@ -504,5 +504,465 @@
             }
         ],
         "timestamp": "2024-12-18T23:29:27.320433"
     }
 ]

             }
         ],
         "timestamp": "2024-12-18T23:29:27.320433"
+    },
+    {
+        "task_id": "59afc37a-0072-44dd-a02a-0cf47d89c120",
+        "model": "ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.6417205190285036,
+        "average_pwed": 0.19048963968896404,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʂiharjoɖarksɯudenɡwisiwaːʂwarɔːjiːr",
+                "per": 0.696969696969697,
+                "pwed": 0.20580808080808083
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "dɔndaːskmidɨkaːɻjɑno̞jwɯräːɡläikθaːn",
+                "per": 0.8214285714285714,
+                "pwed": 0.17338709677419356
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "çizkatːɛnwɔstinanhaːɡɛɾdanɨzbirufubuswɔwoːɾnenʂaːbi",
+                "per": 0.5531914893617021,
+                "pwed": 0.1276595744680851
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðɔriːzɔnsfɾdɔɕtaːivsimtfuøʃnɛu",
+                "per": 0.5862068965517241,
+                "pwed": 0.08764367816091957
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "pɾɔdakʂɔnmɛjfaɔfaɾbuwɔuwɛkspɛktajʂons",
+                "per": 0.7575757575757576,
+                "pwed": 0.18806306306306303
+            }
+        ],
+        "timestamp": "2024-12-19T07:41:18.132953"
+    },
+    {
+        "task_id": "5517f6b2-6a76-4a2d-a6ce-33446f390c3b",
+        "model": "ginic/gender_split_70_female_4_wav2vec2-large-xlsr-buckeye-ipa",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.2810165988557621,
+        "average_pwed": 0.10703377161801164,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʃihædjɹ̩dɑɹksudɪnɡɹisiwɑʃwɑɾɹ̩ɔljiɹ",
+                "per": 0.18181818181818182,
+                "pwed": 0.07196969696969698
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "doʊndæskmitɪkæɹiʌnɔɪliɹæɡlaɪkðæʔ",
+                "per": 0.2857142857142857,
+                "pwed": 0.14062500000000003
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "hɪzkæptʌnwʌzθɪnhæɡɹ̩dɛnɪzbjuɾʌfl̩butswɹ̩wɔʊɹnɪnʃæbi",
+                "per": 0.2978723404255319,
+                "pwed": 0.09114583333333333
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðʌɹizʌnzfɹ̩ðʌstaɪvsimtfulɪʃnaʊ",
+                "per": 0.2413793103448276,
+                "pwed": 0.014367816091954023
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "pɹʌdʌkʃʌnmeɪfɔlfɑɹbʌloʊɛkspɛkteɪʃʌnz",
+                "per": 0.30303030303030304,
+                "pwed": 0.10532407407407407
+            }
+        ],
+        "timestamp": "2024-12-20T13:45:52.009233"
+    },
+    {
+        "task_id": "c2139f96-e79e-4f25-a525-aa039f65555f",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-INTERNATIONAL1.9.2WithoutSpaces",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.9537775908999574,
+        "average_pwed": 0.9351204819224959,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "iɛ2",
+                "per": 0.9696969696969697,
+                "pwed": 0.9406565656565656
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "iɛ2",
+                "per": 0.9285714285714286,
+                "pwed": 0.9285714285714286
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "iɛ2",
+                "per": 0.9787234042553191,
+                "pwed": 0.9583333333333333
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "iɛ2",
+                "per": 0.9655172413793104,
+                "pwed": 0.932471264367816
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "iɛ2",
+                "per": 0.9696969696969697,
+                "pwed": 0.9406565656565656
+            }
+        ],
+        "timestamp": "2024-12-20T14:21:32.290889"
+    },
+    {
+        "task_id": "d146f1f1-6e6e-4b28-9420-c652ae9a1002",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-nl",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.9887075544197294,
+        "average_pwed": 0.9692486915717254,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "p",
+                "per": 1.0,
+                "pwed": 0.9747474747474747
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "p",
+                "per": 1.0,
+                "pwed": 0.96875
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "p",
+                "per": 0.9787234042553191,
+                "pwed": 0.9787234042553191
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "p",
+                "per": 1.0,
+                "pwed": 0.9683908045977011
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "p",
+                "per": 0.9696969696969697,
+                "pwed": 0.9696969696969697
+            }
+        ],
+        "timestamp": "2024-12-20T15:26:27.658798"
+    },
+    {
+        "task_id": "265c5859-e7ba-492d-a6c9-45733dc17c99",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-nl",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.9887075544197294,
+        "average_pwed": 0.9692486915717254,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "p",
+                "per": 1.0,
+                "pwed": 0.9747474747474747
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "p",
+                "per": 1.0,
+                "pwed": 0.96875
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "p",
+                "per": 0.9787234042553191,
+                "pwed": 0.9787234042553191
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "p",
+                "per": 1.0,
+                "pwed": 0.9683908045977011
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "p",
+                "per": 0.9696969696969697,
+                "pwed": 0.9696969696969697
+            }
+        ],
+        "timestamp": "2024-12-20T15:40:51.631218"
+    },
+    {
+        "task_id": "e297dfde-95e5-462b-a6e5-8fa43bc30bc0",
+        "model": "speech31/wavlm-large-english-ipa",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.3694017596969614,
+        "average_pwed": 0.1356824900612308,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʃihædjɔɹdɑɹksutɪnɡɹisiwɑʃwɔtɹ̩ɔljɪɹ",
+                "per": 0.2727272727272727,
+                "pwed": 0.11274509803921567
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "dɑntæskmitəkæɹiænojliɹæɡlajkðæt",
+                "per": 0.39285714285714285,
+                "pwed": 0.13575268817204303
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "hɪzkæpptənwɑzθɪændhæɡɹ̩dænhɪzbjutəfəlbutswɹ̩wɔɹnɪnʃæbi",
+                "per": 0.3404255319148936,
+                "pwed": 0.12980769230769232
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðəɹizənzfɔɹðəsdajvsimdfulɪʃnaw",
+                "per": 0.20689655172413793,
+                "pwed": 0.051388888888888894
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "pɹədʌkʃənmejffɔlfɔɑɹbɪlowɪkspɛktejʃənz",
+                "per": 0.45454545454545453,
+                "pwed": 0.16666666666666666
+            }
+        ],
+        "timestamp": "2024-12-20T16:13:24.050232"
+    },
+    {
+        "task_id": "efe95f71-05e3-485d-8e0c-1823a3037cf4",
+        "model": "speech31/wavlm-large-english-ipa",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.3694017596969614,
+        "average_pwed": 0.1356824900612308,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʃihædjɔɹdɑɹksutɪnɡɹisiwɑʃwɔtɹ̩ɔljɪɹ",
+                "per": 0.2727272727272727,
+                "pwed": 0.11274509803921567
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "dɑntæskmitəkæɹiænojliɹæɡlajkðæt",
+                "per": 0.39285714285714285,
+                "pwed": 0.13575268817204303
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "hɪzkæpptənwɑzθɪændhæɡɹ̩dænhɪzbjutəfəlbutswɹ̩wɔɹnɪnʃæbi",
+                "per": 0.3404255319148936,
+                "pwed": 0.12980769230769232
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðəɹizənzfɔɹðəsdajvsimdfulɪʃnaw",
+                "per": 0.20689655172413793,
+                "pwed": 0.051388888888888894
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "pɹədʌkʃənmejffɔlfɔɑɹbɪlowɪkspɛktejʃənz",
+                "per": 0.45454545454545453,
+                "pwed": 0.16666666666666666
+            }
+        ],
+        "timestamp": "2024-12-20T16:26:47.980084"
+    },
+    {
+        "task_id": "4b2ae2fc-fe2f-4f8b-9e8f-25c0bae13c0d",
+        "model": "speech31/XLS-R-300m-english-ipa",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.36382554692045954,
+        "average_pwed": 0.1299702312124616,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʃihædjɔɹdɑɹksutɪnɡɹisiwɑʃwɔtɹ̩ɔljɪɹ",
+                "per": 0.2727272727272727,
+                "pwed": 0.11274509803921567
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "dɑntæskmitəkæɹiænojliɹæɡlajkðæt",
+                "per": 0.39285714285714285,
+                "pwed": 0.13575268817204303
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "hɪzkæmptənwɑzθɪnændhæɡɹ̩dɪndhɪzbjutəfəlbutswɹ̩wɔɹnɪnʃæbi",
+                "per": 0.3404255319148936,
+                "pwed": 0.14583333333333334
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðəɹɛzənzfɔɹðɪstajvsimdfulɪʃnaw",
+                "per": 0.2413793103448276,
+                "pwed": 0.052777777777777785
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "pɹədʌkʃənmejfɔlfɑɹbɪlowɛkspɛktejʃənz",
+                "per": 0.3939393939393939,
+                "pwed": 0.11921296296296297
+            }
+        ],
+        "timestamp": "2024-12-20T16:47:54.824174"
+    },
+    {
+        "task_id": "33d387c0-703c-415d-b8e2-81cea87a2146",
+        "model": "speech31/wav2vec2-large-english-TIMIT-phoneme_v3",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.44563344149564776,
+        "average_pwed": 0.18844914029048124,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʃihædjʊrdɑrksutɪngrisiwɑʃwɔtərɔljɪrr",
+                "per": 0.3939393939393939,
+                "pwed": 0.12976190476190474
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "doʊntæskmitɪkɛriənɔɪliræglaɪkðətdnt",
+                "per": 0.39285714285714285,
+                "pwed": 0.19730392156862747
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "hɪzkæptənwɑzθɪnəndhægərdəndhɪzbjutəfəlbutswərwɔrnɪnʃæbibæb",
+                "per": 0.44680851063829785,
+                "pwed": 0.20394736842105265
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðərizənzfərðɪsstaɪvsimdfulɪʃnaʊa",
+                "per": 0.27586206896551724,
+                "pwed": 0.11328125
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "prədəkʃənmeɪfɔlfɑrbɪloʊɛkspɛkteɪʃənzd",
+                "per": 0.3939393939393939,
+                "pwed": 0.13626126126126126
+            }
+        ],
+        "timestamp": "2024-12-20T17:05:35.210786"
+    },
+    {
+        "task_id": "c89bcefc-3884-435a-a54c-24297fe6f041",
+        "model": "speech31/wav2vec2-large-TIMIT-IPA2",
+        "subset": "test",
+        "num_files": 1680,
+        "average_per": 0.4847029843149011,
+        "average_pwed": 0.2072006544586948,
+        "detailed_results": [
+            {
+                "file": "data/TEST/DR1/FAKS0/SA1.WAV",
+                "ground_truth": "ʃihædjɹdɑɹksuɾɪŋgɹisiwɑʃwɑɾɹʔɔljiɹ",
+                "prediction": "ʃihædjʊrdɑrksutɪngrisiwɑʃwɔtərɔljɪrər",
+                "per": 0.42424242424242425,
+                "pwed": 0.15393518518518517
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SA2.WAV",
+                "ground_truth": "oʊnæsmitikɛɹiinɔɪliɹæglaɪkðæt",
+                "prediction": "doʊntæskmitɪkɛriənɔɪliræglaɪkðətdoʊndt",
+                "per": 0.5,
+                "pwed": 0.2623873873873874
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI1573.WAV",
+                "ground_truth": "hɪzkæpinwəsθɪnænhægɹdinɪzbjuɾuflbutswɹwɔɹninʃæbi",
+                "prediction": "hɪzkæptənwɑzθɪnəndhægərdəndhɪzbjutəfəlbutswərwɔrnəndʃæbiiii",
+                "per": 0.46808510638297873,
+                "pwed": 0.2191091954022989
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI2203.WAV",
+                "ground_truth": "ðiɹizənzfɹðɪsdaɪvsimdfuliʃnaʊ",
+                "prediction": "ðərizənzfərðɪstaɪvsimdfulɪʃnaʊ",
+                "per": 0.20689655172413793,
+                "pwed": 0.054166666666666675
+            },
+            {
+                "file": "data/TEST/DR1/FAKS0/SI943.WAV",
+                "ground_truth": "ɹdʌkʃinmeɪfɔlfɑɹbəloʊəkspikeɪʃnts",
+                "prediction": "prədəkʃənmeɪfɔlfɑrbɪloʊɛkspɛkteɪʃənzpzppppzpdtdtd",
+                "per": 0.7272727272727273,
+                "pwed": 0.34438775510204084
+            }
+        ],
+        "timestamp": "2024-12-20T22:50:50.641790"
     }
 ]

queue/tasks.json CHANGED Viewed

@@ -124,5 +124,134 @@
         "github_url": "https://huggingface.co/ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
         "status": "completed",
         "submitted_at": "2024-12-18T22:55:36.734691"
     }
 ]

         "github_url": "https://huggingface.co/ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
         "status": "completed",
         "submitted_at": "2024-12-18T22:55:36.734691"
+    },
+    {
+        "id": "59afc37a-0072-44dd-a02a-0cf47d89c120",
+        "model": "ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
+        "subset": "test",
+        "submission_name": "wav2vec2 non-english transcription",
+        "github_url": "https://huggingface.co/ctaguchi/wav2vec2-large-xlsr-japlmthufielta-ipa1000-ns",
+        "status": "completed",
+        "submitted_at": "2024-12-18T23:47:03.488337"
+    },
+    {
+        "id": "e57eda9d-7a1d-4b41-9d47-a3d3839cac8b",
+        "model": "ginic/gender_split_70_female_4_wav2vec2-large-xlsr-buckeye-ipa",
+        "subset": "test",
+        "submission_name": "phonetic transcription with the Buckeye corpus, from xlsr-53 model ",
+        "github_url": "https://github.com/ginic/multipa/tree/buckeye_experiments",
+        "status": "failed",
+        "submitted_at": "2024-12-19T11:48:26.415322",
+        "error": "Evaluation failed: (MaxRetryError(\"HTTPSConnectionPool(host='cdn-lfs-us-1.hf.co', port=443): Max retries exceeded with url: /repos/a4/b1/a4b11f4627350048e021a84d10b89320db54e02c54b2a9366228f8a05cda220b/120f5bc04d1df15143033c93e3ef358981775b529f17e0db11e58a1b80754e67?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model.safetensors%3B+filename%3D%22model.safetensors%22%3B&Expires=1734889736&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTczNDg4OTczNn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zL2E0L2IxL2E0YjExZjQ2MjczNTAwNDhlMDIxYTg0ZDEwYjg5MzIwZGI1NGUwMmM1NGIyYTkzNjYyMjhmOGEwNWNkYTIyMGIvMTIwZjViYzA0ZDFkZjE1MTQzMDMzYzkzZTNlZjM1ODk4MTc3NWI1MjlmMTdlMGRiMTFlNThhMWI4MDc1NGU2Nz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=kfPD6ymEJuVvFZyuN3qL3xk4YJlpI5dqHgON4wJY-Mppwlp6x4Dw7cWdjEkJvMRF-bDuzNWQ3BEJPbsYouVW9WZMucDmxo38UwxSzIBhfWQxCYiHdUWuQPkypDUkI1mR3vbnCFQFXLiMQ2CgwWQz7q66OjIyq3suA00mhL2WcL8wvtovrfoEOkboEXCHCNLprfpoHpfoyfo~VS9~kmm61GN6SWbc9lzASIuT5FLkn~BJ6h405MgutQpNvrR4SHVLftk7rBmY8TAB3re5D0-9qFrMYb2Tk~9RKT3nxSNbgZVcEXzA5rYskcuGsrHoTuTTZ-NSW69K2M0IeivzFWTLNQ__&Key-Pair-Id=K24J24Z295AEI9 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x280544190>: Failed to establish a new connection: [Errno 51] Network is unreachable'))\"), '(Request ID: 14c9cc7c-47ee-47ae-b473-f4add807d233)')"
+    },
+    {
+        "id": "5517f6b2-6a76-4a2d-a6ce-33446f390c3b",
+        "model": "ginic/gender_split_70_female_4_wav2vec2-large-xlsr-buckeye-ipa",
+        "subset": "test",
+        "submission_name": "phonetic transcription with the Buckeye corpus, from xlsr-53 model",
+        "github_url": "https://github.com/ginic/multipa/tree/buckeye_experiments",
+        "status": "completed",
+        "submitted_at": "2024-12-20T13:29:37.327317"
+    },
+    {
+        "id": "c2139f96-e79e-4f25-a525-aa039f65555f",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-INTERNATIONAL1.9.2WithoutSpaces",
+        "subset": "test",
+        "submission_name": "phonetic transcription",
+        "github_url": "https://huggingface.co/Jubliano/wav2vec2-large-xls-r-300m-ipa-INTERNATIONAL1.5WithoutSpaces",
+        "status": "completed",
+        "submitted_at": "2024-12-20T14:01:35.626112"
+    },
+    {
+        "id": "d146f1f1-6e6e-4b28-9420-c652ae9a1002",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-nl",
+        "subset": "test",
+        "submission_name": "Jubliano xlsr model",
+        "github_url": "https://huggingface.co/Jubliano/wav2vec2-large-xls-r-300m-ipa-nl1.1",
+        "status": "completed",
+        "submitted_at": "2024-12-20T15:08:45.949389"
+    },
+    {
+        "id": "265c5859-e7ba-492d-a6c9-45733dc17c99",
+        "model": "Jubliano/wav2vec2-large-xls-r-300m-ipa-nl",
+        "subset": "test",
+        "submission_name": "Jubliano xlsr model",
+        "github_url": "https://huggingface.co/Jubliano/wav2vec2-large-xls-r-300m-ipa-nl1.1",
+        "status": "completed",
+        "submitted_at": "2024-12-20T15:26:27.706187"
+    },
+    {
+        "id": "e297dfde-95e5-462b-a6e5-8fa43bc30bc0",
+        "model": "speech31/wavlm-large-english-ipa",
+        "subset": "test",
+        "submission_name": "speech31 phoneme transcription english",
+        "github_url": "https://huggingface.co/speech31/wavlm-large-english-ipa",
+        "status": "completed",
+        "submitted_at": "2024-12-20T15:56:25.445806"
+    },
+    {
+        "id": "efe95f71-05e3-485d-8e0c-1823a3037cf4",
+        "model": "speech31/wavlm-large-english-ipa",
+        "subset": "test",
+        "submission_name": "speech31 phoneme transcription english",
+        "github_url": "https://huggingface.co/speech31/wavlm-large-english-ipa",
+        "status": "completed",
+        "submitted_at": "2024-12-20T16:13:24.099308"
+    },
+    {
+        "id": "4b2ae2fc-fe2f-4f8b-9e8f-25c0bae13c0d",
+        "model": "speech31/XLS-R-300m-english-ipa",
+        "subset": "test",
+        "submission_name": "speech31 xlsr model",
+        "github_url": "https://huggingface.co/speech31/XLS-R-300m-english-ipa",
+        "status": "completed",
+        "submitted_at": "2024-12-20T16:33:23.864360"
+    },
+    {
+        "id": "33d387c0-703c-415d-b8e2-81cea87a2146",
+        "model": "speech31/wav2vec2-large-english-TIMIT-phoneme_v3",
+        "subset": "test",
+        "submission_name": "model is a fine-tuned version of facebook/wav2vec2-large on the TIMIT dataset",
+        "github_url": "https://huggingface.co/speech31/wav2vec2-large-english-TIMIT-phoneme_v3",
+        "status": "completed",
+        "submitted_at": "2024-12-20T16:52:07.883839"
+    },
+    {
+        "id": "03e4e265-dc1c-4052-88bd-4ef481938d9d",
+        "model": "speech31/wav2vec2-large-TIMIT-IPA2\"",
+        "subset": "test",
+        "submission_name": "fine-tuned version of facebook/wav2vec2-large on the None dataset",
+        "github_url": "https://huggingface.co/speech31/wav2vec2-large-TIMIT-IPA2",
+        "status": "failed",
+        "submitted_at": "2024-12-20T21:54:21.539246",
+        "error": "Evaluation failed: Incorrect path_or_model_id: 'speech31/wav2vec2-large-TIMIT-IPA2\"'. Please provide either the path to a local folder or the repo_id of a model on the Hub."
+    },
+    {
+        "id": "4575245b-ae3c-4969-adef-fd07a58560b9",
+        "model": "speech31/wav2vec2-large-TIMIT-IPA2\"",
+        "subset": "test",
+        "submission_name": "fine-tuned version of facebook/wav2vec2-large on the None dataset",
+        "github_url": "https://huggingface.co/speech31/wav2vec2-large-TIMIT-IPA2",
+        "status": "failed",
+        "submitted_at": "2024-12-20T21:54:25.670276",
+        "error": "Evaluation failed: Incorrect path_or_model_id: 'speech31/wav2vec2-large-TIMIT-IPA2\"'. Please provide either the path to a local folder or the repo_id of a model on the Hub."
+    },
+    {
+        "id": "c89bcefc-3884-435a-a54c-24297fe6f041",
+        "model": "speech31/wav2vec2-large-TIMIT-IPA2",
+        "subset": "test",
+        "submission_name": "fine-tuned version of facebook/wav2vec2-large on the None dataset",
+        "github_url": "https://huggingface.co/speech31/wav2vec2-large-TIMIT-IPA2",
+        "status": "completed",
+        "submitted_at": "2024-12-20T21:54:38.559569"
+    },
+    {
+        "id": "81fa94f8-94ae-4601-952c-24abaddaf691",
+        "model": "ginic/vary_individuals_young_only_3_wav2vec2-large-xlsr-buckeye-ipa",
+        "subset": "test",
+        "submission_name": "ginic model, facebook/wav2vec2-large-xlsr-53 fine tuned",
+        "github_url": "https://huggingface.co/ginic/vary_individuals_young_only_3_wav2vec2-large-xlsr-buckeye-ipa",
+        "status": "processing",
+        "submitted_at": "2024-12-21T01:15:41.870875"
     }
 ]

utils_display.py DELETED Viewed

@@ -1,31 +0,0 @@
-from dataclasses import dataclass
-# These classes are for user facing column names, to avoid having to change them
-# all around the code when a modif is needed
-@dataclass
-class ColumnContent:
-    name: str
-    type: str
-def fields(raw_class):
-    return [v for k, v in raw_class.__dict__.items() if k[:2] != "__" and k[-2:] != "__"]
-@dataclass(frozen=True)
-class AutoEvalColumn: # Auto evals column
-    model = ColumnContent("Model", "markdown")
-    avg_per = ColumnContent("Average PER ⬇️", "number")
-    avg_wped = ColumnContent("Average PWED ⬇️", "number")
-def make_clickable_model(model_name):
-    link = f"https://huggingface.co/{model_name}"
-    return f'<a target="_blank" href="{link}" style="color: var(--link-text-color); text-decoration: underline;text-decoration-style: dotted;">{model_name}</a>'
-def styled_error(error):
-    return f"<p style='color: red; font-size: 20px; text-align: center;'>{error}</p>"
-def styled_warning(warn):
-    return f"<p style='color: orange; font-size: 20px; text-align: center;'>{warn}</p>"
-def styled_message(message):
-    return f"<p style='color: green; font-size: 20px; text-align: center;'>{message}</p>"