RabakBench: Scaling Human Annotations to Construct Localized
  Multilingual Safety Benchmarks for Low-Resource Languages
Large language models (LLMs) and their safety classifiers often perform
poorly on low-resource languages due to limited training data and evaluation
benchmarks. This paper introduces RabakBench, a new multilingual safety
benchmark localized to Singapore's unique linguistic context, covering
Singlish, Chinese, Malay, and Tamil. RabakBench is constructed through a
scalable three-stage pipeline: (i) Generate - adversarial example generation by
augmenting real Singlish web content with LLM-driven red teaming; (ii) Label -
semi-automated multi-label safety annotation using majority-voted LLM labelers
aligned with human judgments; and (iii) Translate - high-fidelity translation
preserving linguistic nuance and toxicity across languages. The final dataset
comprises over 5,000 safety-labeled examples across four languages and six
fine-grained safety categories with severity levels. Evaluations of 11 popular
open-source and closed-source guardrail classifiers reveal significant
performance degradation. RabakBench not only enables robust safety evaluation
in Southeast Asian multilingual settings but also offers a reproducible
framework for building localized safety datasets in low-resource environments.
The benchmark dataset, including the human-verified translations, and
evaluation code are publicly available.
686dfb25cb5725779c60b4c2
Gabriel Chua
claimed_verified
2025-07-09T08:49:41.950Z
655f4ff710e5c5fbef30fd97
https://cdn-avatars.huggingface.co/v1/production/uploads/655f4ff710e5c5fbef30fd97/bI_KQnFof50HRqYBWpyu2.jpeg
Gabriel C
user
gabrielchua
686dfb25cb5725779c60b4c3
Leanne Tan
686dfb25cb5725779c60b4c4
Ziyu Ge
686dfb25cb5725779c60b4c5
Roy Ka-Wei Lee
RabakBench: Scaling Human Annotations to Construct Localized
  Multilingual Safety Benchmarks for Low-Resource Languages
Large language models (LLMs) and their safety classifiers often perform
poorly on low-resource languages due to limited training data and evaluation
benchmarks. This paper introduces RabakBench, a new multilingual safety
benchmark localized to Singapore's unique linguistic context, covering
Singlish, Chinese, Malay, and Tamil. RabakBench is constructed through a
scalable three-stage pipeline: (i) Generate - adversarial example generation by
augmenting real Singlish web content with LLM-driven red teaming; (ii) Label -
semi-automated multi-label safety annotation using majority-voted LLM labelers
aligned with human judgments; and (iii) Translate - high-fidelity translation
preserving linguistic nuance and toxicity across languages. The final dataset
comprises over 5,000 safety-labeled examples across four languages and six
fine-grained safety categories with severity levels. Evaluations of 11 popular
open-source and closed-source guardrail classifiers reveal significant
performance degradation. RabakBench not only enables robust safety evaluation
in Southeast Asian multilingual settings but also offers a reproducible
framework for building localized safety datasets in low-resource environments.
The benchmark dataset, including the human-verified translations, and
evaluation code are publicly available.
RabakBench is a multilingual safety benchmark for Singapore's languages, enabling robust safety evaluation and dataset creation in low-resource environments.
adversarial example generation LLM-driven red teaming semi-automated multi-label safety annotation majority-voted LLM labelers high-fidelity translation safety-labeled examples fine-grained safety categories guardrail classifiers reproducible framework localized safety datasets