SE Dereverberation SB 16kHz Small
Model Overview
Description
The model extracts speech for human or machine listeners. This is a generative speech dereverberation model based on the Schrödinger bridge. The model is trained on a publicly available research dataset.
This model is for research and development only.
License/Terms of Use
License to use this model is covered by the CC-BY-NC-SA-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-NC-SA-4.0 license.
References
[1] Schrödinger Bridge for Generative Speech Enhancement, Interspeech, 2024.
Model Architecture
Architecture Type: Schrödinger Bridge
Network Architecture: U-Net with convolutional layers
Input
Input Type(s): Audio
Input Format(s): .wav files
Input Parameters: One-Dimensional (1D)
Other Properties Related to Input: 16000 Hz Mono-channel Audio
Output
Output Type(s): Audio
Output Format: .wav files
Output Parameters: One-Dimensional (1D)
Other Properties Related to Output: 16000 Hz Mono-channel Audio
Software Integration
Runtime Engine(s):
- NeMo-2.0.0
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Jetson
- NVIDIA Hopper
- NVIDIA Lovelace
- NVIDIA Turing
- NVIDIA Volta
Preferred Operating System(s)
- Linux
- Windows
Model Version(s)
se_der_sb_16k_small_v1.0
Training, Testing, and Evaluation Datasets
Training Dataset
Link: WSJ0
Data Collection Method by dataset: Human
Labeling Method by dataset: Human
Properties (Quantity, Dataset Descriptions, Sensor(s)):
WSJ0 was used for clean speech signals. The observed signals are simulated with room impulse responses with reverberation times between 0.4 seconds and 1.0 seconds, and without any background noise. The total size of the training dataset was approximately 25 hours.
Testing Dataset
Link: WSJ0
Data Collection Method by dataset: Human
Labeling Method by dataset: Human
Properties (Quantity, Dataset Descriptions, Sensor(s)):
WSJ0 was used for clean speech signals. The observed signals are simulated with room impulse responses with reverberation times between 0.4 seconds and 1.0 seconds, and without any background noise. The total size of the training dataset was approximately 2 hours.
Evaluation Dataset
Link: WSJ0
Data Collection Method by dataset: Human
Labeling Method by dataset: Human
Properties (Quantity, Dataset Descriptions, Sensor(s)):
WSJ0 was used for clean speech signals. The observed signals are simulated with room impulse responses with reverberation times between 0.4 seconds and 1.0 seconds, and without any background noise. The total size of the training dataset was approximately 2 hours.
Inference
Engine: NeMo 2.0
Test Hardware: NVIDIA v100
Performance
The model is trained on the training subset of the WSJ0-Reverb dataset using the auxiliary L1-norm loss [1].
The model is evaluated using several instrumental metrics: perceptual evaluation of speech quality (PESQ), extended short-term objective intelligibility (ESTOI) and scale-invariant signal-to-distortion ratio (SI-SDR). Word error rate (WER) is evaluated using the FastConformer-Transducer-Large English ASR model.
Metrics are reported on the test set of WSJ0-Reverb dataset using either SDE or ODE sampler.
| Signal | PESQ | ESTOI | SI-SDR/dB | WER / % |
|---|---|---|---|---|
| Input | 1.29 | 0.44 | -9.5 | 8.29 |
| Processed SDE | 2.79 | 0.89 | 7.4 | 4.27 |
| Processed ODE | 2.59 | 0.86 | 6.2 | 5.79 |
How to use this model
The model is available for use in the NVIDIA NeMo toolkit, and can be used to process audio or for fine-tuning.
Load the model
from nemo.collections.audio.models import AudioToAudioModel
model = AudioToAudioModel.from_pretrained('nvidia/se_der_sb_16k_small')
Process audio
A single audio file can be processed as follows
import librosa
audio_in, _ = librosa.load(path_to_input_audio, sr=model.sample_rate)
audio_in_signal = torch.from_numpy(audio_in).view(1, 1, -1).to(device)
audio_in_length = torch.tensor([audio_in_signal.size(-1)]).to(device)
audio_out_signal, _ = model(input_signal=audio_in_signal, input_length=audio_in_length)
For processing several audio files at once, check the process_audio script in NeMo.
Listen to audio
import soundfile as sf
audio_out = audio_out_signal.cpu().numpy().squeeze()
sf.write(path_to_output_audio, audio_out, samplerate=model.sample_rate)
Change sampler configuration
model.sampler.process = 'ode' # default sampler is 'sde'
model.sampler.num_steps = 10 # default is 50 steps
audio_out_signal, _ = model(input_signal=audio_in_signal, input_length=audio_in_length)
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report security vulnerabilities or NVIDIA AI Concerns here.