--- license: mit library_name: pytorch tags: - speech-enhancement - audio - denoising - onnx - causal - streaming - real-time - edge-ai - stm32 datasets: - JacobLinCool/VoiceBank-DEMAND-16k --- # LiSenNet Ultra-compact, causal, real-time speech enhancers trained on VoiceBank-DEMAND-16k — a sub-band U-Net with a magnitude-only mask (phase from a 2-iteration Griffin-Lim offline, or the noisy phase for real-time). Port of **Yan, Zhou, Chen & Lu, _LiSenNet_, [arXiv:2409.13285](https://arxiv.org/abs/2409.13285)** ([hyyan2k/LiSenNet](https://github.com/hyyan2k/LiSenNet), MIT). This repo holds **four variants**, each in its own subfolder: | subfolder | recipe | params | NPU-compiles | FP32 PESQ | real-time int8 PESQ | | --------- | ------ | -----: | :----------: | --------: | ------------------: | | [`gru/`](./gru) | dual-path **GRU** (faithful) | 36,783 | ✗ | 3.006 | 2.930 | | [`conv/`](./conv) | dual-path **conv** | 41,063 | ✗ | 2.970 | 2.855 | | [`conv-hardened/`](./conv-hardened) | conv + **NPU-hardened** | 36,288 | ✓ | 3.013 | 2.998 | | [`conv-hardened-deep/`](./conv-hardened-deep) | hardened + **deep RF + ReLU6** | 46,248 | ✓* | **3.084** | **3.014** (int8) / **3.052** (hybrid) | PESQ is wideband, on the full 824-utterance VoiceBank-DEMAND test split. (*) `conv-hardened-deep/` uses the same op set as `conv-hardened/` plus Clip (ReLU6); its graph has not yet been through a stedgeai compile, `conv-hardened/` has (topology verified on Neural-ART). * **`gru/`** is the faithful reproduction and the original quality reference. Its GRU + 2-axis `LayerNorm` do **not** compile to the STM32N6 Neural-ART NPU. * **`conv/`** replaces the GRU bottleneck with a dual-path conv one (0 GRU / 0 LayerNormalization). Its ops map to the NPU, but the FIFO-state streaming graph (`conv/g_best_streaming_fp32.onnx`, `feat + N state_i_in -> est_mag + N state_i_out`) crashes the Neural-ART codegen — kept as the CPU/onnxruntime frame-by-frame reference. * **`conv-hardened/`** is the compile-verified **NPU-deployable** variant: per-channel BatchNorm (folds into the convs), ReLU, plain ConvTranspose upsampling, and a stateless **windowed** deploy graph (`conv-hardened/g_best_windowed_int8_static.onnx`, signed QInt8, `feat_window (B,3,132,257) -> est_mag (B,64,257)`, window = receptive field 68 + 64 emitted frames) that **compiles to Neural-ART** — the artifact handed to stedgeai. The hardened primitives also quantize far better (int8 drop −0.016 vs −0.115 for `conv/`). It also ships a **frame-by-frame streaming** graph — see below. * **`conv-hardened-deep/`** is the **best model overall**: the hardened recipe, deeper (3 blocks) with an extra dilation stage (receptive field 196 frames ≈ 3.1 s) and **ReLU6** activations (bounded ranges quantize better; exports as Clip). Window is 196+64=260 frames (`feat_window (B,3,260,257)`). It ships **two** signed windowed int8 artifacts: `g_best_windowed_int8_static.onnx` (everything int8, PESQ 3.014) and `g_best_windowed_int8_decoder_fp32.onnx` (int8 except the decoder's QDQ nodes, PESQ **3.052** — the int8 loss is decoder-localized; the decoder then runs as float epochs on the board). ## Two deploy paths (`conv-hardened/`) — throughput vs latency The hardened variant ships **both** NPU-deployable graphs. They are the same trained weights; they differ only in how much audio each inference call consumes. Both are measured on an STM32N6570-DK (STM32N657, MCU 800 MHz / NPU 1 GHz, stedgeai 4.0.1); PESQ is the full 824-utterance VBD test split, int8 + noisy phase. | graph | shape | algorithmic latency | latency/frame | RTF | PESQ | | ----- | ----- | ------------------- | ------------: | --: | ---: | | `g_best_windowed_int8_static.onnx` | `feat_window (B,3,132,257) -> est_mag (B,64,257)` | 1.02 s block | **1.15 ms** | 0.072 | **2.998** | | `g_best_streaming_int8_static.onnx` | `feat (B,3,1,257) + 17 states -> est_mag (B,1,257) + 17 states` | **one 16 ms hop** | 2.79 ms | 0.174 | **2.982** | * the **windowed** graph is stateless and the throughput champion (1.15 ms per emitted frame), but buffers 1.02 s of audio per call — use it when block latency is acceptable; * the **streaming** graph is true frame-by-frame — one 16 ms hop in, one enhanced frame out, with 17 bounded FIFO state tensors carried on-device (47 KiB weights + 147 KiB activations, entirely on-chip). It costs ~2.4× more compute per frame but drops algorithmic latency by ~64×, which makes it the best-quality *streaming* speech enhancer measured on this NPU. Both are signed QInt8 (the Neural-ART recipe). The streaming graph is calibrated by threading the real FIFO state through the trained model, so the state tensors see realistic ranges; its int8 cost is −0.018 PESQ vs the same-protocol FP32 streaming reference (3.000). Code + full write-up: [https://github.com/LarocheC/eco8-neaixt](https://github.com/LarocheC/eco8-neaixt) — see [RESULTS_LISENNET.md](https://github.com/LarocheC/eco8-neaixt/blob/main/RESULTS_LISENNET.md). ## Files (per subfolder) `config.json`, `g_best` (PyTorch `{"generator": state_dict}`), `g_best_fp32.onnx` and `g_best_int8_static.onnx` (whole-utterance mask sub-network, `feat (B,3,T,F) -> est_mag (B,T,F)`). `conv/` additionally has `g_best_streaming_fp32.onnx` and `g_best_streaming_int8_static.onnx` (single frame + explicit state I/O — CPU/onnxruntime reference only; this variant's streaming graph does not compile to Neural-ART). `conv-hardened/` has **both** NPU deploy graphs: `g_best_windowed_{fp32,int8_static}.onnx` (stateless windowed) and `g_best_streaming_{fp32,int8_static}.onnx` (frame-by-frame, 17 FIFO states) — see the two-deploy-paths table above. The ONNX graphs are the mask sub-network only — STFT, feature build and phase recovery stay host-side. ## Loading (PyTorch) ```python import json, torch from huggingface_hub import hf_hub_download from common.env import AttrDict from lisennet.model import build_lisennet REPO, SUB = "claroche1/LiSenNet", "conv-hardened" # or "gru" / "conv" cfg = json.load(open(hf_hub_download(REPO, f"{SUB}/config.json"))) ckpt = torch.load(hf_hub_download(REPO, f"{SUB}/g_best"), map_location="cpu", weights_only=True) model = build_lisennet(AttrDict(cfg)).eval() model.load_state_dict(ckpt["generator"]) # model(noisy_wav)["est"] ``` ## Running the NPU windowed deploy graph (`conv-hardened/`) Stateless: feed a sliding window of the last `68 + 64 = 132` feature frames and read the 64 newest enhanced-magnitude frames (no state tensors to carry). ```python import numpy as np, onnxruntime as ort from huggingface_hub import hf_hub_download sess = ort.InferenceSession( hf_hub_download("claroche1/LiSenNet", "conv-hardened/g_best_windowed_int8_static.onnx"), providers=["CPUExecutionProvider"], ) feat_window = np.zeros((1, 3, 132, 257), np.float32) # last 68+64 feature frames est_mag = sess.run(["est_mag"], {"feat_window": feat_window})[0] # (1, 64, 257) ``` ## Running the streaming graph frame-by-frame (`conv-hardened/`, `conv/`) One 16 ms hop in, one enhanced frame out; the FIFO state tensors are threaded from each call's outputs into the next call's inputs (start-of-stream = zeros). Point it at `conv-hardened/` for the NPU-deployable graph (PESQ 2.982), or `conv/` for the CPU-only reference. ```python import numpy as np, onnxruntime as ort from huggingface_hub import hf_hub_download sess = ort.InferenceSession( hf_hub_download("claroche1/LiSenNet", "conv-hardened/g_best_streaming_int8_static.onnx"), providers=["CPUExecutionProvider"], ) state_in = [i for i in sess.get_inputs() if i.name != "feat"] # FIFO states out_names = [o.name for o in sess.get_outputs()] # est_mag + state_*_out zeros = lambda s: np.zeros([d if isinstance(d, int) else 1 for d in s], np.float32) states = {i.name: zeros(i.shape) for i in state_in} # start-of-stream = zeros def step(feat_t): # feat_t: (1, 3, 1, 257) res = sess.run(out_names, {"feat": feat_t, **states}) for i, v in zip(state_in, res[1:]): states[i.name] = v return res[0] # est_mag (1, 1, 257) ``` ## License MIT. See the [source repository](https://github.com/LarocheC/eco8-neaixt) for training code and full attribution.