Snowflake/snowflake-arctic-embed-l-v2.0
Sentence Similarity
•
Updated
•
22.2k
•
80
Gabriel C
gabrielchua
AI & ML interests
Large Language Models, AI Safety, Causal Inference
Recent Activity
liked
a model
about 4 hours ago
Snowflake/snowflake-arctic-embed-l-v2.0
liked
a dataset
2 days ago
allenai/WildChat-1M-Full
liked
a dataset
2 days ago
allenai/WildChat-1M
Organizations
gabrielchua's activity
liked
a
model
about 4 hours ago
liked
2
datasets
2 days ago
liked
a
Space
12 days ago
upvoted
a
paper
14 days ago
liked
a
Space
16 days ago
updated
a
Space
19 days ago
upvoted
a
paper
21 days ago
updated
a
Space
21 days ago
liked
a
dataset
24 days ago
liked
a
model
25 days ago
liked
2
datasets
25 days ago
updated
a
dataset
25 days ago
upvoted
3
collections
28 days ago
updated
a
Space
30 days ago
posted
an
update
about 1 month ago
Post
1208
Sharing my first paper!
==
Large Language Models (LLMs) are powerful, but they're prone to off-topic misuse, where users push them beyond their intended scope. Think harmful prompts, jailbreaks, and misuse. So how do we build better guardrails?
Traditional guardrails rely on curated examples or classifiers. The problem?
⚠️ High false-positive rates
⚠️ Poor adaptability to new misuse types
⚠️ Require real-world data, which is often unavailable during pre-production
Our method skips the need for real-world misuse examples. Instead, we:
1️⃣ Define the problem space qualitatively
2️⃣ Use an LLM to generate synthetic misuse prompts
3️⃣ Train and test guardrails on this dataset
We apply this to the off-topic prompt detection problem, and fine-tune simple bi- and cross-encoder classifiers that outperform heuristics based on cosine similarity or prompt engineering.
Additionally, framing the problem as prompt relevance allows these fine-tuned classifiers to generalise to other risk categories (e.g., jailbreak, toxic prompts).
Through this work, we also open-source our dataset (2M examples, ~50M+ tokens) and models.
paper: A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection (2411.12946)
artifacts: govtech/off-topic-guardrail-673838a62e4c661f248e81a4
==
Large Language Models (LLMs) are powerful, but they're prone to off-topic misuse, where users push them beyond their intended scope. Think harmful prompts, jailbreaks, and misuse. So how do we build better guardrails?
Traditional guardrails rely on curated examples or classifiers. The problem?
⚠️ High false-positive rates
⚠️ Poor adaptability to new misuse types
⚠️ Require real-world data, which is often unavailable during pre-production
Our method skips the need for real-world misuse examples. Instead, we:
1️⃣ Define the problem space qualitatively
2️⃣ Use an LLM to generate synthetic misuse prompts
3️⃣ Train and test guardrails on this dataset
We apply this to the off-topic prompt detection problem, and fine-tune simple bi- and cross-encoder classifiers that outperform heuristics based on cosine similarity or prompt engineering.
Additionally, framing the problem as prompt relevance allows these fine-tuned classifiers to generalise to other risk categories (e.g., jailbreak, toxic prompts).
Through this work, we also open-source our dataset (2M examples, ~50M+ tokens) and models.
paper: A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection (2411.12946)
artifacts: govtech/off-topic-guardrail-673838a62e4c661f248e81a4
upvoted
a
paper
about 1 month ago