VocalBurst Classifier — multi-label
A presence classifier for non-speech vocal bursts (laughs, sighs, gasps, coughs, groans, throat
sounds, …) over the 82-class LAION VocalBurst taxonomy + no_burst. Multi-label (independent
sigmoid per class) — a clip can contain several bursts. Works on pure bursts and bursts embedded in speech.
Architecture
laion/voiceclap-commercial (768-d, frozen) → MLP 2048-wide × 4 deep (LayerNorm+GELU+dropout) → 83 sigmoids. ~14.4M trainable params.
Training
- Positives: Gemini-confirmed single bursts (DACVAE, 82 classes) + real speech-with-bursts (gemini_pro_25, multi-label).
- Negatives / speech carriers: Emilia clips filtered burst-free by
laion/vocalburst-locator. - ~97k samples: pure bursts + composites (burst + speech carriers + a 2nd burst) + 25% overlays (burst mixed over speech) + neg+neg. The speech carriers carry no label, so the head must localise the burst, not memorise audio. Balanced (≤1000/class); 50 epochs, AdamW, BCE with per-class pos-weight, best-mAP checkpoint.
v2 (current): retrained with 9,000 multilingual burst-free negatives (FLEURS: Chinese, Hindi, Bengali, Arabic, Persian, Urdu, Tamil, Telugu, Vietnamese, Thai, Indonesian, Japanese, Korean, Swahili, Yoruba, Zulu, Turkish, Russian) — fixes hallucinated bursts on non-European speech.
Results (Gemini-mixed validation)
| metric | fine (82) | coarse (16 families) |
|---|---|---|
| macro mAP | 0.39 | 0.64 |
| top-1 exact | 49% | – |
| a true label in top-3 | 77% | – |
| Best when you take the top-1 prediction (which the two-stage combo demo uses). Much stronger at the coarse "which family of sound" level than the exact fine subtype. |
How it works
Two stages: a frozen laion/voiceclap-commercial audio encoder turns a clip into a 768-d embedding
(encode_waveform, auto-downloaded — the repo needs no extra setup), then this small trained MLP head
maps it to 83 outputs = 82 VocalBurst classes + no_burst (taxonomy: LAION-AI/voice-taxonomies · vocalburst).
A no-burst gate: if P(no_burst) ≥ 0.5 the clip is declared burst-free (no false alarm); otherwise the top classes are returned. Clips are truncated to the first 30 s (the encoder's window).
Usage
from inference import VocalBurstClassifier
clf = VocalBurstClassifier("laion/vocalburst-classifier-multilabel") # HF repo id, or a local checkout dir
print(clf.predict("clip.wav")) # -> {no_burst, p_no_burst, top1, predictions:[(class,prob)], group}
Files
model.pt (MLP weights) · config.json (arch) · classes.json (83 labels, index order) ·
class_to_group.json (fine→16 coarse families) · inference.py · example.py · requirements.txt.
Interactive demos (audio + predictions)
- Multi-label predictions vs ground truth: https://projects.laion.ai/procedural-voice-captions/vocalburst-predictions/
- Single-burst classifier: https://projects.laion.ai/procedural-voice-captions/vocalburst-single/
- Two-stage detect→classify combo: https://projects.laion.ai/procedural-voice-captions/vocalburst-combo/
- On MOSS character voices: https://projects.laion.ai/procedural-voice-captions/vocalburst-character/
Training data + embeddings: laion/vocalburst-classification. Embedder: laion/voiceclap-commercial. License: CC-BY-4.0.
- Downloads last month
- 42