A Flexible Large Language Models Guardrail Development Methodology
  Applied to Off-Topic Prompt Detection
Large Language Models are prone to off-topic misuse, where users may prompt
these models to perform tasks beyond their intended scope. Current guardrails,
which often rely on curated examples or custom classifiers, suffer from high
false-positive rates, limited adaptability, and the impracticality of requiring
real-world data that is not available in pre-production. In this paper, we
introduce a flexible, data-free guardrail development methodology that
addresses these challenges. By thoroughly defining the problem space
qualitatively and passing this to an LLM to generate diverse prompts, we
construct a synthetic dataset to benchmark and train off-topic guardrails that
outperform heuristic approaches. Additionally, by framing the task as
classifying whether the user prompt is relevant with respect to the system
prompt, our guardrails effectively generalize to other misuse categories,
including jailbreak and harmful prompts. Lastly, we further contribute to the
field by open-sourcing both the synthetic dataset and the off-topic guardrail
models, providing valuable resources for developing guardrails in
pre-production environments and supporting future research and development in
LLM safety.
673e9a33ed6ba7ea687cb7ae
Gabriel Chua
claimed_verified
2024-11-21T08:09:39.171Z
655f4ff710e5c5fbef30fd97
https://cdn-avatars.huggingface.co/v1/production/uploads/655f4ff710e5c5fbef30fd97/bI_KQnFof50HRqYBWpyu2.jpeg
Gabriel C
user
gabrielchua
673e9a33ed6ba7ea687cb7af
Shing Yee Chan
extracted_pending
2024-11-21T02:25:56.265Z
66e39d4311e4f40aa760f063
/avatars/1d95e1ab088caeb4366f052fd42d426d.svg
Chan Shing Yee
user
shingurding
673e9a33ed6ba7ea687cb7b0
Shaun Khoo
claimed_verified
2024-11-25T20:39:23.848Z
6564650eb1181111eaa992f8
/avatars/d7e8503baff5513c2d6f8f7c48608563.svg
Shaun Khoo
user
shaunkhoo
A Flexible Large Language Models Guardrail Development Methodology
  Applied to Off-Topic Prompt Detection
Large Language Models are prone to off-topic misuse, where users may prompt
these models to perform tasks beyond their intended scope. Current guardrails,
which often rely on curated examples or custom classifiers, suffer from high
false-positive rates, limited adaptability, and the impracticality of requiring
real-world data that is not available in pre-production. In this paper, we
introduce a flexible, data-free guardrail development methodology that
addresses these challenges. By thoroughly defining the problem space
qualitatively and passing this to an LLM to generate diverse prompts, we
construct a synthetic dataset to benchmark and train off-topic guardrails that
outperform heuristic approaches. Additionally, by framing the task as
classifying whether the user prompt is relevant with respect to the system
prompt, our guardrails effectively generalize to other misuse categories,
including jailbreak and harmful prompts. Lastly, we further contribute to the
field by open-sourcing both the synthetic dataset and the off-topic guardrail
models, providing valuable resources for developing guardrails in
pre-production environments and supporting future research and development in
LLM safety.
A flexible methodology for creating off-topic guardrails in LLMs uses synthetic datasets generated by LLMs to improve accuracy and generalization over heuristic approaches.
Large Language Models off-topic misuse guardrails synthetic dataset heuristic approaches jailbreak harmful prompts