add

Browse files

Files changed (16) hide show

README.md +98 -52
config.yaml +1 -1
reproducible_research/2022.07/AISHELL.SpeakerDiarization.Full.test.eval +24 -0
reproducible_research/2022.07/AISHELL.SpeakerDiarization.Full.test.rttm +0 -0
reproducible_research/2022.07/AMI-SDM.SpeakerDiarization.only_words.test.eval +20 -0
reproducible_research/2022.07/AMI-SDM.SpeakerDiarization.only_words.test.rttm +0 -0
reproducible_research/2022.07/AMI.SpeakerDiarization.only_words.test.eval +20 -0
reproducible_research/2022.07/AMI.SpeakerDiarization.only_words.test.rttm +0 -0
reproducible_research/2022.07/CALLHOME.SpeakerDiarization.CALLHOME.test.eval +254 -0
reproducible_research/2022.07/CALLHOME.SpeakerDiarization.CALLHOME.test.rttm +0 -0
reproducible_research/2022.07/DIHARD.SpeakerDiarization.Full.test.eval +263 -0
reproducible_research/2022.07/DIHARD.SpeakerDiarization.Full.test.rttm +0 -0
reproducible_research/2022.07/REPERE.SpeakerDiarization.Full.test.eval +65 -0
reproducible_research/2022.07/REPERE.SpeakerDiarization.Full.test.rttm +0 -0
reproducible_research/2022.07/VoxConverse.SpeakerDiarization.VoxConverse.test.eval +236 -0
reproducible_research/2022.07/VoxConverse.SpeakerDiarization.VoxConverse.test.rttm +0 -0

README.md CHANGED Viewed

@@ -1,6 +1,5 @@
 ---
-license: mit
-tags:
 - pyannote
 - pyannote-audio
 - pyannote-audio-pipeline
@@ -10,76 +9,123 @@ tags:
 - speaker
 - speaker-diarization
 - speaker-change-detection
-- endpoints-template
-library_name: generic
 ---
-# WIP: Depends on [#1098](https://github.com/pyannote/pyannote-audio/pull/1098)
-# 🎹 Speaker diarization with Pyannote and Inference Endpoints
-This repository implements a custom `handler` for `speaker-diarization` for 🤗 Inference Endpoints using Pyannote. The code for the customized pipeline is in the [handler.py](https://huggingface.co/philschmid/pyannote-speaker-diarization-endpoint/blob/main/handler.py).
-There is also a [notebook](https://huggingface.co/philschmid/pyannote-speaker-diarization-endpoint/blob/main/create_handler.ipynb) included, on how to create the `handler.py`
-###  Request
-The endpoint expects a binary audio file. Below are a cURL and a Python example using the `requests` library.
-**curl**
-```bash
-# load audio file
-wget https://cdn-media.huggingface.co/speech_samples/sample1.flac
-# run request
-curl --request POST \
-  --url https://{ENDPOINT}/ \
-  --header 'Content-Type: audio/x-wav' \
-  --header 'Authorization: Bearer {HF_TOKEN}' \
-  --data-binary '@sample.wav'
 ```
-**Python**
 ```python
-import json
-from typing import List
-import requests as r
-import base64
-import mimetypes
-ENDPOINT_URL=""
-HF_TOKEN=""
-def predict(path_to_audio:str=None):
-    # read audio file
-    with open(path_to_audio, "rb") as i:
-      b = i.read()
-    # get mimetype
-    content_type= mimetypes.guess_type(path_to_audio)[0]
-    headers= {
-        "Authorization": f"Bearer {HF_TOKEN}",
-        "Content-Type": content_type
-    }
-    response = r.post(ENDPOINT_URL, headers=headers, data=b)
-    return response.json()
-prediction = predict(path_to_audio="sample.wav")
-prediction
-```
-expected output
-```json
-{"diarization": [
-{"label": "SPEAKER_01", "start": "0.4978125", "stop": "1.3921875"},
-{"label": "SPEAKER_01", "start": "1.8984375", "stop": "2.7590624999999998"},
-{"label": "SPEAKER_02", "start": "2.9953125", "stop": "3.5015625000000004"},
-{"label": "SPEAKER_01", "start": "3.5690625000000002", "stop": "4.311562500000001"}
-...
 ```

 ---
+tags:
 - pyannote
 - pyannote-audio
 - pyannote-audio-pipeline
 - speaker
 - speaker-diarization
 - speaker-change-detection
+- voice-activity-detection
+- overlapped-speech-detection
+- automatic-speech-recognition
+datasets:
+- ami
+- dihard
+- voxconverse
+- aishell
+- repere
+- voxceleb
+license: mit
 ---
+# 🎹 Speaker diarization
+Relies on pyannote.audio 2.0: see [installation instructions](https://github.com/pyannote/pyannote-audio/tree/develop#installation).
+## TL;DR
+```python
+# load the pipeline from Hugginface Hub
+from pyannote.audio import Pipeline
+pipeline = Pipeline.from_pretrained("pyannote/[email protected]")
+# apply the pipeline to an audio file
+diarization = pipeline("audio.wav")
+# dump the diarization output to disk using RTTM format
+with open("audio.rttm", "w") as rttm:
+    diarization.write_rttm(rttm)
+```
+## Advanced usage
+In case the number of speakers is known in advance, one can use the `num_speakers` option:
+```python
+diarization = pipeline("audio.wav", num_speakers=2)
+```
+One can also provide lower and/or upper bounds on the number of speakers using `min_speakers` and `max_speakers` options:
+```python
+diarization = pipeline("audio.wav", min_speakers=2, max_speakers=5)
 ```
+If you feel adventurous, you can try and play with the various pipeline hyper-parameters.
+For instance, one can use a more aggressive voice activity detection by increasing the value of `segmentation_onset` threshold:
 ```python
+hparams = pipeline.parameters(instantiated=True)
+hparams["segmentation_onset"] += 0.1
+pipeline.instantiate(hparams)
+```
+## Benchmark
+### Real-time factor
+Real-time factor is around 5% using one Nvidia Tesla V100 SXM2 GPU (for the neural inference part) and one Intel Cascade Lake 6248 CPU (for the clustering part).
+In other words, it takes approximately 3 minutes to process a one hour conversation.
+### Accuracy
+This pipeline is benchmarked on a growing collection of datasets.
+Processing is fully automatic:
+* no manual voice activity detection (as is sometimes the case in the literature)
+* no manual number of speakers (though it is possible to provide it to the pipeline)
+* no fine-tuning of the internal models nor tuning of the pipeline hyper-parameters to each dataset
+... with the least forgiving diarization error rate (DER) setup (named *"Full"* in [this paper](https://doi.org/10.1016/j.csl.2021.101254)):
+* no forgiveness collar
+* evaluation of overlapped speech
+| Benchmark                                                                                                                          | [DER%](. "Diarization error rate") | [FA%](. "False alarm rate") | [Miss%](. "Missed detection rate") | [Conf%](. "Speaker confusion rate") | Expected output                                                                            | File-level evaluation                                                                      |
+| ---------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------- | ---------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
+| [AISHELL-4](http://www.openslr.org/111/)                                                                                           | 14.61                              | 3.31                        | 4.35                               | 6.95                                | [RTTM](reproducible_research/AISHELL.SpeakerDiarization.Full.test.rttm)                    | [eval](reproducible_research/AISHELL.SpeakerDiarization.Full.test.eval)                    |
+| [AMI *Mix-Headset*](https://groups.inf.ed.ac.uk/ami/corpus/) [*only_words*](https://github.com/BUTSpeechFIT/AMI-diarization-setup) | 18.21                              | 3.28                        | 11.07                              | 3.87                                | [RTTM](reproducible_research/2022.07/AMI.SpeakerDiarization.only_words.test.rttm)          | [eval](reproducible_research/2022.07/AMI.SpeakerDiarization.only_words.test.eval)          |
+| [AMI *Array1-01*](https://groups.inf.ed.ac.uk/ami/corpus/) [*only_words*](https://github.com/BUTSpeechFIT/AMI-diarization-setup)   | 29.00                              | 2.71                        | 21.61                              | 4.68                                | [RTTM](reproducible_research/2022.07/AMI-SDM.SpeakerDiarization.only_words.test.rttm)      | [eval](reproducible_research/2022.07/AMI-SDM.SpeakerDiarization.only_words.test.eval)      |
+| [CALLHOME](https://catalog.ldc.upenn.edu/LDC2001S97) [*Part2*](https://github.com/BUTSpeechFIT/CALLHOME_sublists/issues/1)         | 30.24                              | 3.71                        | 16.86                              | 9.66                                | [RTTM](reproducible_research/2022.07/CALLHOME.SpeakerDiarization.CALLHOME.test.rttm)       | [eval](reproducible_research/2022.07/CALLHOME.SpeakerDiarization.CALLHOME.test.eval)       |
+| [DIHARD 3 *Full*](https://arxiv.org/abs/2012.01477)                                                                                | 20.99                              | 4.25                        | 10.74                              | 6.00                                | [RTTM](reproducible_research/2022.07/DIHARD.SpeakerDiarization.Full.test.rttm)             | [eval](reproducible_research/2022.07/DIHARD.SpeakerDiarization.Full.test.eval)             |
+| [REPERE *Phase 2*](https://islrn.org/resources/360-758-359-485-0/)                                                                 | 12.62                              | 1.55                        | 3.30                               | 7.76                                | [RTTM](reproducible_research/2022.07/REPERE.SpeakerDiarization.Full.test.rttm)             | [eval](reproducible_research/2022.07/REPERE.SpeakerDiarization.Full.test.eval)             |
+| [VoxConverse *v0.0.2*](https://github.com/joonson/voxconverse)                                                                     | 12.76                              | 3.45                        | 3.85                               | 5.46                                | [RTTM](reproducible_research/2022.07/VoxConverse.SpeakerDiarization.VoxConverse.test.rttm) | [eval](reproducible_research/2022.07/VoxConverse.SpeakerDiarization.VoxConverse.test.eval) |
+## Support
+For commercial enquiries and scientific consulting, please contact [me](mailto:[email protected]).
+For [technical questions](https://github.com/pyannote/pyannote-audio/discussions) and [bug reports](https://github.com/pyannote/pyannote-audio/issues), please check [pyannote.audio](https://github.com/pyannote/pyannote-audio) Github repository.
+## Citations
+```bibtex
+@inproceedings{Bredin2021,
+  Title = {{End-to-end speaker segmentation for overlap-aware resegmentation}},
+  Author = {{Bredin}, Herv{\'e} and {Laurent}, Antoine},
+  Booktitle = {Proc. Interspeech 2021},
+  Address = {Brno, Czech Republic},
+  Month = {August},
+  Year = {2021},
+}
 ```
+```bibtex
+@inproceedings{Bredin2020,
+  Title = {{pyannote.audio: neural building blocks for speaker diarization}},
+  Author = {{Bredin}, Herv{\'e} and {Yin}, Ruiqing and {Coria}, Juan Manuel and {Gelly}, Gregory and {Korshunov}, Pavel and {Lavechin}, Marvin and {Fustes}, Diego and {Titeux}, Hadrien and {Bouaziz}, Wassim and {Gill}, Marie-Philippe},
+  Booktitle = {ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing},
+  Address = {Barcelona, Spain},
+  Month = {May},
+  Year = {2020},
+}
+```

config.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 pipeline:
   name: pyannote.audio.pipelines.SpeakerDiarization
   params:
-    segmentation: philschmid/pyannote-speaker-diarization-endpoint
     segmentation_step: 0.1
     segmentation_batch_size: 32
     # embedding: speechbrain/spkrec-ecapa-voxceleb@5c0be3875fda05e81f3c004ed8c7c06be308de1e

 pipeline:
   name: pyannote.audio.pipelines.SpeakerDiarization
   params:
+    segmentation: pyannote/[email protected]
     segmentation_step: 0.1
     segmentation_batch_size: 32
     # embedding: speechbrain/spkrec-ecapa-voxceleb@5c0be3875fda05e81f3c004ed8c7c06be308de1e

reproducible_research/2022.07/AISHELL.SpeakerDiarization.Full.test.eval ADDED Viewed

	@@ -0,0 +1,24 @@

+             diarization error rate    total  correct correct false alarm false alarm missed detection missed detection confusion confusion
+                                  %                         %                       %                                 %                   %
+item
+L_R003S01C02                  18.89  2311.97  1980.25   85.65      104.95        4.54           144.55             6.25    187.17      8.10
+L_R003S02C02                  12.74  2036.79  1832.36   89.96       55.15        2.71            93.48             4.59    110.94      5.45
+L_R003S03C02                  16.46  2376.45  2045.30   86.07       59.97        2.52           206.09             8.67    125.06      5.26
+L_R003S04C02                  17.63  2011.85  1752.94   87.13       95.70        4.76            78.03             3.88    180.89      8.99
+L_R004S01C01                  10.96  2041.84  1877.74   91.96       59.67        2.92            53.53             2.62    110.58      5.42
+L_R004S02C01                  14.78  2204.89  1970.28   89.36       91.27        4.14            57.61             2.61    177.01      8.03
+L_R004S03C01                  12.31  2314.18  2103.86   90.91       74.52        3.22            86.45             3.74    123.87      5.35
+L_R004S06C01                   9.26  2029.95  1861.81   91.72       19.85        0.98            44.98             2.22    123.15      6.07
+M_R003S01C01                  15.04  2191.47  1935.43   88.32       73.45        3.35           126.17             5.76    129.87      5.93
+M_R003S02C01                  15.14  2376.79  2052.14   86.34       35.24        1.48           204.97             8.62    119.69      5.04
+M_R003S04C01                  37.43  2015.63  1444.81   71.68      183.54        9.11            93.33             4.63    477.50     23.69
+M_R003S05C01                  46.60  2249.68  1344.71   59.77      143.29        6.37           134.14             5.96    770.84     34.26
+S_R003S01C01                  11.02  2127.86  1984.73   93.27       91.34        4.29            69.86             3.28     73.27      3.44
+S_R003S02C01                   8.70  2222.87  2097.15   94.34       67.66        3.04            71.94             3.24     53.78      2.42
+S_R003S03C01                  10.01  2071.12  1957.05   94.49       93.19        4.50            71.56             3.45     42.51      2.05
+S_R003S04C01                  16.47  2615.78  2254.19   86.18       69.33        2.65           218.44             8.35    143.15      5.47
+S_R004S01C01                   4.65  2117.91  2043.76   96.50       24.30        1.15            64.34             3.04      9.81      0.46
+S_R004S02C01                   5.04  2095.08  2027.67   96.78       38.13        1.82            35.96             1.72     31.45      1.50
+S_R004S03C01                   2.13  2074.39  2046.66   98.66       16.48        0.79            26.72             1.29      1.01      0.05
+S_R004S04C01                   4.43  2078.85  2029.99   97.65       43.19        2.08            12.66             0.61     36.20      1.74
+TOTAL                         14.61 43565.36 38642.83   88.70     1440.22        3.31          1894.78             4.35   3027.75      6.95

reproducible_research/2022.07/AISHELL.SpeakerDiarization.Full.test.rttm ADDED Viewed