add

README.md

@@ -7,126 +7,85 @@ tasks:
 pipeline_tag: voice-activity-detection
 ---
 
-# FSMN-Monophone VAD 模型介绍
 
-[//]: # (FSMN-Monophone VAD 模型)
+# FunASR: A Fundamental End-to-End Speech Recognition Toolkit
 
-## Highlight
-- 16k中文通用VAD模型：可用于检测长语音片段中有效语音的起止时间点。
-  - 基于[Paraformer-large长音频模型](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)场景的使用
-  - 基于[FunASR框架](https://github.com/alibaba-damo-academy/FunASR)，可进行ASR，VAD，[中文标点](https://www.modelscope.cn/models/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch/summary)的自由组合
-  - 基于音频数据的有效语音片段起止时间点检测
 
-## <strong>[FunASR开源项目介绍](https://github.com/alibaba-damo-academy/FunASR)</strong>
-<strong>[FunASR](https://github.com/alibaba-damo-academy/FunASR)</strong>希望在语音识别的学术研究和工业应用之间架起一座桥梁。通过发布工业级语音识别模型的训练和微调，研究人员和开发人员可以更方便地进行语音识别模型的研究和生产，并推动语音识别生态的发展。让语音识别更有趣！
+[![PyPI](https://img.shields.io/pypi/v/funasr)](https://pypi.org/project/funasr/)
 
-[**github仓库**](https://github.com/alibaba-damo-academy/FunASR)
-| [**最新动态**](https://github.com/alibaba-damo-academy/FunASR#whats-new) 
-| [**环境安装**](https://github.com/alibaba-damo-academy/FunASR#installation)
-| [**服务部署**](https://www.funasr.com)
-| [**模型库**](https://github.com/alibaba-damo-academy/FunASR/tree/main/model_zoo)
-| [**联系我们**](https://github.com/alibaba-damo-academy/FunASR#contact)
 
+<strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！
 
-## 模型原理介绍
+[**Highlights**](#highlights)
+| [**News**](https://github.com/alibaba-damo-academy/FunASR#whats-new) 
+| [**Installation**](#installation)
+| [**Quick Start**](#quick-start)
+| [**Runtime**](./runtime/readme.md)
+| [**Model Zoo**](#model-zoo)
+| [**Contact**](#contact)
 
-FSMN-Monophone VAD是达摩院语音团队提出的高效语音端点检测模型，用于检测输入音频中有效语音的起止时间点信息，并将检测出来的有效音频片段输入识别引擎进行识别，减少无效语音带来的识别错误。
 
-<p align="center">
-<img src="fig/struct.png" alt="VAD模型结构"  width="500" />
+<a name="highlights"></a>
+## Highlights
+- FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models.
+- We have released a vast collection of academic and industrial pretrained models on the [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition) and [huggingface](https://huggingface.co/FunASR), which can be accessed through our [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md). The representative [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the [service deployment document](runtime/readme_cn.md). 
 
-FSMN-Monophone VAD模型结构如上图所示：模型结构层面，FSMN模型结构建模时可考虑上下文信息，训练和推理速度快，且时延可控；同时根据VAD模型size以及低时延的要求，对FSMN的网络结构、右看帧数进行了适配。在建模单元层面，speech信息比较丰富，仅用单类来表征学习能力有限，我们将单一speech类升级为Monophone。建模单元细分，可以避免参数平均，抽象学习能力增强，区分性更好。
 
-## 基于ModelScope进行推理
+<a name="Installation"></a>
+## Installation
 
-- 推理支持音频格式如下：
-  - wav文件路径，例如：data/test/audios/vad_example.wav
-  - wav文件url，例如：https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav
-  - wav二进制数据，格式bytes，例如：用户直接从文件里读出bytes数据或者是麦克风录出bytes数据。
-  - 已解析的audio音频，例如：audio, rate = soundfile.read("vad_example_zh.wav")，类型为numpy.ndarray或者torch.Tensor。
-  - wav.scp文件，需符合如下要求：
-
-```sh
-cat wav.scp
-vad_example1  data/test/audios/vad_example1.wav
-vad_example2  data/test/audios/vad_example2.wav
-...
+```shell
+pip3 install -U funasr
 ```
-
-- 若输入格式wav文件url，api调用方式可参考如下范例：
-
-```python
-from modelscope.pipelines import pipeline
-from modelscope.utils.constant import Tasks
-
-inference_pipeline = pipeline(
-    task=Tasks.voice_activity_detection,
-    model='iic/speech_fsmn_vad_zh-cn-16k-common-pytorch',
-    model_revision="v2.0.4",
-)
-
-segments_result = inference_pipeline(input='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav')
-print(segments_result)
+Or install from source code
+``` sh
+git clone https://github.com/alibaba/FunASR.git && cd FunASR
+pip3 install -e ./
 ```
+Install modelscope for the pretrained models (Optional)
 
-- 输入音频为pcm格式，调用api时需要传入音频采样率参数fs，例如：
-
-```python
-segments_result = inference_pipeline(input='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.pcm', fs=16000)
+```shell
+pip3 install -U modelscope
 ```
 
-- 若输入格式为文件wav.scp(注：文件名需要以.scp结尾)，可添加 output_dir 参数将识别结果写入文件中，参考示例如下：
+## Model Zoo
+FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo]().
 
-```python
-inference_pipeline(input="wav.scp", output_dir='./output_dir')
-```
-识别结果输出路径结构如下：
+(Note: 🤗 represents the Huggingface model zoo link, ⭐ represents the ModelScope model zoo link)
 
-```sh
-tree output_dir/
-output_dir/
-└── 1best_recog
-    └── text
 
-1 directory, 1 files
-```
-text：VAD检测语音起止时间点结果文件（单位：ms）
+|                                                                             Model Name                                                                             |                    Task Details                    |          Training Data           | Parameters |
+|:------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------:|:--------------------------------:|:----------:|
+|    paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)  [🤗]() )    | speech recognition, with timestamps, non-streaming |      60000 hours, Mandarin       |    220M    |
+| <nobr>paraformer-zh-streaming <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗]() )</nobr> |           speech recognition, streaming            |      60000 hours, Mandarin       |    220M    |
+|         paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗]() )         | speech recognition, with timestamps, non-streaming |       50000 hours, English       |    220M    |
+|                     conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗]() )                      |         speech recognition, non-streaming          |       50000 hours, English       |    220M    |
+|                     ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗]() )                      |              punctuation restoration               |    100M, Mandarin and English    |    1.1G    | 
+|                          fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗]() )                          |              voice activity detection              | 5000 hours, Mandarin and English |    0.4M    | 
+|                          fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗]() )                           |                timestamp prediction                |       5000 hours, Mandarin       |    38M     | 
+|                cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗]() )                                             |        speaker verification/diarization            |            5000 hours            |    7.2M    | 
 
-- 若输入音频为已解析的audio音频，api调用方式可参考如下范例：
 
-```python
-import soundfile
 
-waveform, sample_rate = soundfile.read("vad_example_zh.wav")
-segments_result = inference_pipeline(input=waveform)
-print(segments_result)
-```
-
-- VAD常用参数调���说明（参考：vad.yaml文件）：
-  - max_end_silence_time：尾部连续检测到多长时间静音进行尾点判停，参数范围500ms～6000ms，默认值800ms(该值过低容易出现语音提前截断的情况)。
-  - speech_noise_thres：speech的得分减去noise的得分大于此值则判断为speech，参数范围：（-1,1）
-    - 取值越趋于-1，噪音被误判定为语音的概率越大，FA越高
-    - 取值越趋于+1，语音被误判定为噪音的概率越大，Pmiss越高
-    - 通常情况下，该值会根据当前模型在长语音测试集上的效果取balance
-    
 
+[//]: # ()
+[//]: # (FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs]&#40;funasr/runtime/docs/SDK_tutorial.md&#41;. More detailed information about service deployment can be found in the [deployment roadmap]&#40;funasr/runtime/readme_cn.md&#41;.)
 
 
-## 基于FunASR进行推理
+<a name="quick-start"></a>
+## Quick Start
 
-下面为快速上手教程，测试音频（[中文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav)，[英文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav)）
+Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English]()).
 
-### 可执行命令行
-在命令行终端执行：
+### Command-line usage
 
 ```shell
-funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++input=vad_example.wav
+funasr +model=paraformer-zh +vad_model="fsmn-vad" +punc_model="ct-punc" +input=asr_example_zh.wav
 ```
 
-注：支持单条音频文件识别，也支持文件列表，列表为kaldi风格wav.scp：`wav_id   wav_path`
+Notes: Support recognition of single audio file, as well as file list in Kaldi-style wav.scp format: `wav_id wav_pat`
 
-### python示例
-#### 非实时语音识别
+### Speech Recognition (Non-streaming)
 ```python
 from funasr import AutoModel
 # paraformer-zh is a multi-functional asr model
@@ -137,14 +96,13 @@ model = AutoModel(model="paraformer-zh", model_revision="v2.0.4",
                   # spk_model="cam++", spk_model_revision="v2.0.2",
                   )
 res = model.generate(input=f"{model.model_path}/example/asr_example.wav", 
-            batch_size_s=300, 
-            hotword='魔搭')
+                     batch_size_s=300, 
+                     hotword='魔搭')
 print(res)
 ```
-注：`model_hub`：表示模型仓库，`ms`为选择modelscope下载，`hf`为选择huggingface下载。
-
-#### 实时语音识别
+Note: `model_hub`: represents the model repository, `ms` stands for selecting ModelScope download, `hf` stands for selecting Huggingface download.
 
+### Speech Recognition (Streaming)
 ```python
 from funasr import AutoModel
 
@@ -169,21 +127,18 @@ for i in range(total_chunk_num):
     res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back)
     print(res)
 ```
+Note: `chunk_size` is the configuration for streaming latency.` [0,10,5]` indicates that the real-time display granularity is `10*60=600ms`, and the lookahead information is `5*60=300ms`. Each inference input is `600ms` (sample points are `16000*0.6=960`), and the output is the corresponding text. For the last speech segment input, `is_final=True` needs to be set to force the output of the last word.
 
-注：`chunk_size`为流式延时配置，`[0,10,5]`表示上屏实时出字粒度为`10*60=600ms`，未来信息为`5*60=300ms`。每次推理输入为`600ms`（采样点数为`16000*0.6=960`），输出为对应文字，最后一个语音片段输入需要设置`is_final=True`来强制输出最后一个字。
-
-#### 语音端点检测（非实时）
+### Voice Activity Detection (Non-Streaming)
 ```python
 from funasr import AutoModel
 
 model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")
-
 wav_file = f"{model.model_path}/example/asr_example.wav"
 res = model.generate(input=wav_file)
 print(res)
 ```
-
-#### 语音端点检测（实时）
+### Voice Activity Detection (Streaming)
 ```python
 from funasr import AutoModel
 
@@ -205,57 +160,23 @@ for i in range(total_chunk_num):
     if len(res[0]["value"]):
         print(res)
 ```
-
-#### 标点恢复
+### Punctuation Restoration
 ```python
 from funasr import AutoModel
 
 model = AutoModel(model="ct-punc", model_revision="v2.0.4")
-
 res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")
 print(res)
 ```
-
-#### 时间戳预测
+### Timestamp Prediction
 ```python
 from funasr import AutoModel
 
 model = AutoModel(model="fa-zh", model_revision="v2.0.4")
-
 wav_file = f"{model.model_path}/example/asr_example.wav"
 text_file = f"{model.model_path}/example/text.txt"
 res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
 print(res)
 ```
 
-更多详细用法（[示例](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)）
-
-
-## 微调
-
-详细用法（[示例](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)）
-
-
-
-
-
-## 使用方式以及适用范围
-
-运行范围
-- 支持Linux-x86_64、Mac和Windows运行。
-
-使用方式
-- 直接推理：可以直接对长语音数据进行计算，有效语音片段的起止时间点信息（单位：ms）。
-
-## 相关论文以及引用信息
-
-```BibTeX
-@inproceedings{zhang2018deep,
-  title={Deep-FSMN for large vocabulary continuous speech recognition},
-  author={Zhang, Shiliang and Lei, Ming and Yan, Zhijie and Dai, Lirong},
-  booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
-  pages={5869--5873},
-  year={2018},
-  organization={IEEE}
-}
-```
+More examples ref to [docs](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)