Ferret: Faster and Effective Automated Red Teaming with Reward-Based
  Scoring Technique
In today's era, where large language models (LLMs) are integrated into
numerous real-world applications, ensuring their safety and robustness is
crucial for responsible AI usage. Automated red-teaming methods play a key role
in this process by generating adversarial attacks to identify and mitigate
potential vulnerabilities in these models. However, existing methods often
struggle with slow performance, limited categorical diversity, and high
resource demands. While Rainbow Teaming, a recent approach, addresses the
diversity challenge by framing adversarial prompt generation as a
quality-diversity search, it remains slow and requires a large fine-tuned
mutator for optimal performance. To overcome these limitations, we propose
Ferret, a novel approach that builds upon Rainbow Teaming by generating
multiple adversarial prompt mutations per iteration and using a scoring
function to rank and select the most effective adversarial prompt. We explore
various scoring functions, including reward models, Llama Guard, and
LLM-as-a-judge, to rank adversarial mutations based on their potential harm to
improve the efficiency of the search for harmful mutations. Our results
demonstrate that Ferret, utilizing a reward model as a scoring function,
improves the overall attack success rate (ASR) to 95%, which is 46% higher than
Rainbow Teaming. Additionally, Ferret reduces the time needed to achieve a 90%
ASR by 15.2% compared to the baseline and generates adversarial prompts that
are transferable i.e. effective on other LLMs of larger size. Our codes are
available at https://github.com/declare-lab/ferret.
66c59d26ed6dabdeafc2a897
Tej Deep Pala
claimed_verified
2025-02-11T10:04:42.037Z
633f875450d83f5065d459d6
/avatars/48312867ac2c05d589869972f90881b9.svg
Pala Tej Deep
user
Tej3
66c59d26ed6dabdeafc2a898
Vernon Y. H. Toh
66c59d26ed6dabdeafc2a899
Rishabh Bhardwaj
claimed_verified
2024-08-22T19:50:41.601Z
5f278507e923d665e616271b
https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
Rishabh Bhardwaj
user
RishabhBhardwaj
66c59d26ed6dabdeafc2a89a
Soujanya Poria
claimed_verified
2024-08-22T19:50:44.235Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
Soujanya Poria
user
soujanyaporia
Ferret: Faster and Effective Automated Red Teaming with Reward-Based
  Scoring Technique
In today's era, where large language models (LLMs) are integrated into
numerous real-world applications, ensuring their safety and robustness is
crucial for responsible AI usage. Automated red-teaming methods play a key role
in this process by generating adversarial attacks to identify and mitigate
potential vulnerabilities in these models. However, existing methods often
struggle with slow performance, limited categorical diversity, and high
resource demands. While Rainbow Teaming, a recent approach, addresses the
diversity challenge by framing adversarial prompt generation as a
quality-diversity search, it remains slow and requires a large fine-tuned
mutator for optimal performance. To overcome these limitations, we propose
Ferret, a novel approach that builds upon Rainbow Teaming by generating
multiple adversarial prompt mutations per iteration and using a scoring
function to rank and select the most effective adversarial prompt. We explore
various scoring functions, including reward models, Llama Guard, and
LLM-as-a-judge, to rank adversarial mutations based on their potential harm to
improve the efficiency of the search for harmful mutations. Our results
demonstrate that Ferret, utilizing a reward model as a scoring function,
improves the overall attack success rate (ASR) to 95%, which is 46% higher than
Rainbow Teaming. Additionally, Ferret reduces the time needed to achieve a 90%
ASR by 15.2% compared to the baseline and generates adversarial prompts that
are transferable i.e. effective on other LLMs of larger size. Our codes are
available at https://github.com/declare-lab/ferret.
Ferret is a novel approach that enhances the efficiency and effectiveness of adversarial prompt generation for large language models by using multiple mutations and a reward model scoring function.
LLMs adversarial attacks automated red-teaming quality-diversity search Rainbow Teaming Ferret adversarial prompt mutations scoring function reward models Llama Guard LLM-as-a-judge attack success rate transferable adversarial prompts