|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- google/cvss |
|
|
language: |
|
|
- en |
|
|
- fr |
|
|
metrics: |
|
|
- bleu |
|
|
--- |
|
|
# NAST-S2X: A Fast and End-to-End Simultaneous Speech-to-Any Translation Model |
|
|
<p align="center"> |
|
|
<img src="https://github.com/ictnlp/NAST-S2x/assets/43530347/02d6dea6-5887-459e-9938-bc510b6c850c"/> |
|
|
</p> |
|
|
|
|
|
## Features |
|
|
* 🤖 **An end-to-end model without intermediate text decoding** |
|
|
* 💪 **Supports offline and streaming decoding of all modalities** |
|
|
* ⚡️ **28× faster inference compared to autoregressive models** |
|
|
|
|
|
## Examples |
|
|
#### We present an example of French-to-English translation using chunk sizes of 320 ms, 2560 ms, and in offline conditions. |
|
|
* Generation with chunk sizes of 320 ms and 2560 ms starts generating English translation before the source speech is complete. |
|
|
* In the examples of simultaneous interpretation, the left audio channel is the input streaming speech, and the right audio channel is the simultaneous translation. |
|
|
> [!NOTE] |
|
|
> For a better experience, please wear headphones. |
|
|
|
|
|
|Chunk Size 320ms | Chunk Size 2560ms | Offline| |
|
|
:-------------------------:|:-------------------------: |:-------------------------: |
|
|
<video src="https://github.com/ictnlp/NAST-S2x/assets/43530347/52f2d5c4-43ad-49cb-844f-09575ef048e0"></video> | <video src="https://github.com/ictnlp/NAST-S2x/assets/43530347/56475dee-1649-40d9-9cb6-9fe033f6bb32"></video> | <video src="https://github.com/ictnlp/NAST-S2x/assets/43530347/b6fb1d09-b418-45f0-84e9-e6ed3a2cea48"></video> |
|
|
|
|
|
Source Speech Transcript | Reference Text Translation |
|
|
:-------------------------:|:-------------------------: |
|
|
Avant la fusion des communes, Rouge-Thier faisait partie de la commune de Louveigné.| before the fusion of the towns rouge thier was a part of the town of louveigne |
|
|
|
|
|
> [!NOTE] |
|
|
> For more examples, please check https://nast-s2x.github.io/. |
|
|
|
|
|
## Performance |
|
|
|
|
|
* ⚡️ **Lightning Fast**: 28× faster inference and competitive quality in offline speech-to-speech translation |
|
|
* 👩💼 **Simultaneous**: Achieves high-quality simultaneous interpretation within a delay of less than 3 seconds |
|
|
* 🤖 **Unified Framework**: Support end-to-end text & speech generation in one model |
|
|
|
|
|
|
|
|
**Check Details** 👇 |
|
|
| Offline-S2S | |
|
|
:-------------------------: |
|
|
| |
|
|
|
|
|
| Simul-S2S | Simul-S2T| |
|
|
:-------------------------:|:-------------------------: |
|
|
 |  |
|
|
|
|
|
|
|
|
## Architecture |
|
|
<p align="center"> |
|
|
<img src="https://github.com/ictnlp/NAST-S2x/assets/43530347/404cdd56-a9d9-4c10-96aa-64f0c7605248" width="800" /> |
|
|
</p> |
|
|
|
|
|
* **Fully Non-autoregressive:** Trained with **CTC-based non-monotonic latent alignment loss [(Shao and Feng, 2022)](https://arxiv.org/abs/2210.03953)** and **glancing mechanism [(Qian et al., 2021)](https://arxiv.org/abs/2008.07905)**. |
|
|
* **Minimum Human Design:** Seamlessly switch between offline translation and simultaneous interpretation **by adjusting the chunk size**. |
|
|
* **End-to-End:** Generate target speech **without** target text decoding. |
|
|
|
|
|
# Sources and Usage |
|
|
## Model |
|
|
> [!NOTE] |
|
|
> We release French-to-English speech-to-speech translation models trained on the CVSS-C dataset to reproduce results in our paper. You can train models in your desired languages by following the instructions provided below. |
|
|
|
|
|
[🤗 Model card](https://huggingface.co/ICTNLP/NAST-S2X) |
|
|
| Chunk Size | checkpoint | ASR-BLEU | ASR-BLEU (Silence Removed) | Average Lagging | |
|
|
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |---------------------------------------------------------------- | |
|
|
| 320ms | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/chunk_320ms.pt) | 19.67 | 24.90 | -393ms | |
|
|
| 1280ms | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/chunk_1280ms.pt) | 20.20 | 25.71 | 3330ms | |
|
|
| 2560ms | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/chunk_2560ms.pt) | 24.88 | 26.14 | 4976ms | |
|
|
| Offline | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/Offline.pt) | 25.82 | - | - | |
|
|
|
|
|
| Vocoder | |
|
|
| --- | |
|
|
|<p align="center"> [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/tree/main/vocoder)</p>| |
|
|
|
|
|
## Inference |
|
|
> [!WARNING] |
|
|
> Before executing all the provided shell scripts, please ensure to replace the variables in the file with the paths specific to your machine. |
|
|
|
|
|
### Offline Inference |
|
|
* **Data preprocessing**: Follow the instructions in the [document](https://github.com/ictnlp/NAST-S2x/blob/main/Preprocessing.md). |
|
|
* **Generate Acoustic Unit**: Excute [``offline_s2u_infer.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/test_scripts/offline_s2u_infer.sh) |
|
|
* **Generate Waveform**: Excute [``offline_wav_infer.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/test_scripts/offline_wav_infer.sh) |
|
|
* **Evaluation**: Using Fairseq's [ASR-BLEU evaluation toolkit](https://github.com/facebookresearch/fairseq/tree/main/examples/speech_to_speech/asr_bleu) |
|
|
### Simultaneous Inference |
|
|
* We use our customized fork of [``SimulEval: b43a7c``](https://github.com/Paulmzr/SimulEval/tree/b43a7c7a9f20bb4c2ff48cf1bc573b4752d7081e) to evaluate the model in simultaneous inference. This repository is built upon the official [``SimulEval: a1435b``](https://github.com/facebookresearch/SimulEval/tree/a1435b65331cac9d62ea8047fe3344153d7e7dac) and includes additional latency scorers. |
|
|
* **Data preprocessing**: Follow the instructions in the [document](https://github.com/ictnlp/NAST-S2x/blob/main/Preprocessing.md). |
|
|
* **Streaming Generation and Evaluation**: Excute [``streaming_infer.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/test_scripts/streaming_infer.sh) |
|
|
|
|
|
## Train your own NAST-S2X |
|
|
* **Data preprocessing**: Follow the instructions in the [document](https://github.com/ictnlp/NAST-S2x/blob/main/Preprocessing.md). |
|
|
* **CTC Pretraining**: Excute [``train_ctc.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/train_scripts/train_ctc.sh) |
|
|
* **NMLA Training**: Excute [``train_nmla.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/train_scripts/train_nmla.sh) |
|
|
|
|
|
## Citing |
|
|
|
|
|
Please kindly cite us if you find our papers or codes useful. |
|
|
|
|
|
``` |
|
|
@inproceedings{ |
|
|
ma2024nonautoregressive, |
|
|
title={A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation}, |
|
|
author={Ma, Zhengrui and Fang, Qingkai and Zhang, Shaolei and Guo, Shoutao and Feng, Yang and Zhang, Min |
|
|
}, |
|
|
booktitle={Proceedings of ACL 2024}, |
|
|
year={2024}, |
|
|
} |
|
|
|
|
|
@inproceedings{ |
|
|
fang2024ctcs2ut, |
|
|
title={CTC-based Non-autoregressive Textless Speech-to-Speech Translation}, |
|
|
author={Fang, Qingkai and Ma, Zhengrui and Zhou, Yan and Zhang, Min and Feng, Yang |
|
|
}, |
|
|
booktitle={Findings of ACL 2024}, |
|
|
year={2024}, |
|
|
} |
|
|
``` |
