title: A Flexible Large Language Models Guardrail Development Methodology
    Applied to Off-Topic Prompt Detection
summary: Large Language Models are prone to off-topic misuse, where users may prompt
  these models to perform tasks beyond their intended scope. Current guardrails,
  which often rely on curated examples or custom classifiers, suffer from high
  false-positive rates, limited adaptability, and the impracticality of requiring
  real-world data that is not available in pre-production. In this paper, we
  introduce a flexible, data-free guardrail development methodology that
  addresses these challenges. By thoroughly defining the problem space
  qualitatively and passing this to an LLM to generate diverse prompts, we
  construct a synthetic dataset to benchmark and train off-topic guardrails that
  outperform heuristic approaches. Additionally, by framing the task as
  classifying whether the user prompt is relevant with respect to the system
  prompt, our guardrails effectively generalize to other misuse categories,
  including jailbreak and harmful prompts. Lastly, we further contribute to the
  field by open-sourcing both the synthetic dataset and the off-topic guardrail
  models, providing valuable resources for developing guardrails in
  pre-production environments and supporting future research and development in
  LLM safety.
paper_authors._id: 673e9a33ed6ba7ea687cb7ae
paper_authors.name: Gabriel Chua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:39.171Z
paper_authors._id: 655f4ff710e5c5fbef30fd97
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655f4ff710e5c5fbef30fd97/bI_KQnFof50HRqYBWpyu2.jpeg
paper_authors.fullname: Gabriel C
paper_authors.type: user
paper_authors.user: gabrielchua
paper_authors._id: 673e9a33ed6ba7ea687cb7af
paper_authors.name: Shing Yee Chan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-21T02:25:56.265Z
paper_authors._id: 66e39d4311e4f40aa760f063
paper_authors.avatarUrl: /avatars/1d95e1ab088caeb4366f052fd42d426d.svg
paper_authors.fullname: Chan Shing Yee
paper_authors.type: user
paper_authors.user: shingurding
paper_authors._id: 673e9a33ed6ba7ea687cb7b0
paper_authors.name: Shaun Khoo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T20:39:23.848Z
paper_authors._id: 6564650eb1181111eaa992f8
paper_authors.avatarUrl: /avatars/d7e8503baff5513c2d6f8f7c48608563.svg
paper_authors.fullname: Shaun Khoo
paper_authors.type: user
paper_authors.user: shaunkhoo
paper_title: A Flexible Large Language Models Guardrail Development Methodology
    Applied to Off-Topic Prompt Detection
paper_summary: Large Language Models are prone to off-topic misuse, where users may prompt
  these models to perform tasks beyond their intended scope. Current guardrails,
  which often rely on curated examples or custom classifiers, suffer from high
  false-positive rates, limited adaptability, and the impracticality of requiring
  real-world data that is not available in pre-production. In this paper, we
  introduce a flexible, data-free guardrail development methodology that
  addresses these challenges. By thoroughly defining the problem space
  qualitatively and passing this to an LLM to generate diverse prompts, we
  construct a synthetic dataset to benchmark and train off-topic guardrails that
  outperform heuristic approaches. Additionally, by framing the task as
  classifying whether the user prompt is relevant with respect to the system
  prompt, our guardrails effectively generalize to other misuse categories,
  including jailbreak and harmful prompts. Lastly, we further contribute to the
  field by open-sourcing both the synthetic dataset and the off-topic guardrail
  models, providing valuable resources for developing guardrails in
  pre-production environments and supporting future research and development in
  LLM safety.