title: Ferret: Faster and Effective Automated Red Teaming with Reward-Based
    Scoring Technique
summary: In today's era, where large language models (LLMs) are integrated into
  numerous real-world applications, ensuring their safety and robustness is
  crucial for responsible AI usage. Automated red-teaming methods play a key role
  in this process by generating adversarial attacks to identify and mitigate
  potential vulnerabilities in these models. However, existing methods often
  struggle with slow performance, limited categorical diversity, and high
  resource demands. While Rainbow Teaming, a recent approach, addresses the
  diversity challenge by framing adversarial prompt generation as a
  quality-diversity search, it remains slow and requires a large fine-tuned
  mutator for optimal performance. To overcome these limitations, we propose
  Ferret, a novel approach that builds upon Rainbow Teaming by generating
  multiple adversarial prompt mutations per iteration and using a scoring
  function to rank and select the most effective adversarial prompt. We explore
  various scoring functions, including reward models, Llama Guard, and
  LLM-as-a-judge, to rank adversarial mutations based on their potential harm to
  improve the efficiency of the search for harmful mutations. Our results
  demonstrate that Ferret, utilizing a reward model as a scoring function,
  improves the overall attack success rate (ASR) to 95%, which is 46% higher than
  Rainbow Teaming. Additionally, Ferret reduces the time needed to achieve a 90%
  ASR by 15.2% compared to the baseline and generates adversarial prompts that
  are transferable i.e. effective on other LLMs of larger size. Our codes are
  available at https://github.com/declare-lab/ferret.
paper_authors._id: 66c59d26ed6dabdeafc2a897
paper_authors.name: Tej Deep Pala
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:04:42.037Z
paper_authors._id: 633f875450d83f5065d459d6
paper_authors.avatarUrl: /avatars/48312867ac2c05d589869972f90881b9.svg
paper_authors.fullname: Pala Tej Deep
paper_authors.type: user
paper_authors.user: Tej3
paper_authors._id: 66c59d26ed6dabdeafc2a898
paper_authors.name: Vernon Y. H. Toh
paper_authors._id: 66c59d26ed6dabdeafc2a899
paper_authors.name: Rishabh Bhardwaj
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:41.601Z
paper_authors._id: 5f278507e923d665e616271b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
paper_authors.fullname: Rishabh Bhardwaj
paper_authors.type: user
paper_authors.user: RishabhBhardwaj
paper_authors._id: 66c59d26ed6dabdeafc2a89a
paper_authors.name: Soujanya Poria
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:44.235Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: Ferret: Faster and Effective Automated Red Teaming with Reward-Based
    Scoring Technique
paper_summary: In today's era, where large language models (LLMs) are integrated into
  numerous real-world applications, ensuring their safety and robustness is
  crucial for responsible AI usage. Automated red-teaming methods play a key role
  in this process by generating adversarial attacks to identify and mitigate
  potential vulnerabilities in these models. However, existing methods often
  struggle with slow performance, limited categorical diversity, and high
  resource demands. While Rainbow Teaming, a recent approach, addresses the
  diversity challenge by framing adversarial prompt generation as a
  quality-diversity search, it remains slow and requires a large fine-tuned
  mutator for optimal performance. To overcome these limitations, we propose
  Ferret, a novel approach that builds upon Rainbow Teaming by generating
  multiple adversarial prompt mutations per iteration and using a scoring
  function to rank and select the most effective adversarial prompt. We explore
  various scoring functions, including reward models, Llama Guard, and
  LLM-as-a-judge, to rank adversarial mutations based on their potential harm to
  improve the efficiency of the search for harmful mutations. Our results
  demonstrate that Ferret, utilizing a reward model as a scoring function,
  improves the overall attack success rate (ASR) to 95%, which is 46% higher than
  Rainbow Teaming. Additionally, Ferret reduces the time needed to achieve a 90%
  ASR by 15.2% compared to the baseline and generates adversarial prompts that
  are transferable i.e. effective on other LLMs of larger size. Our codes are
  available at https://github.com/declare-lab/ferret.
paper_ai_summary: Ferret is a novel approach that enhances the efficiency and effectiveness of adversarial prompt generation for large language models by using multiple mutations and a reward model scoring function.
paper_ai_keywords: LLMs adversarial attacks automated red-teaming quality-diversity search Rainbow Teaming Ferret adversarial prompt mutations scoring function reward models Llama Guard LLM-as-a-judge attack success rate transferable adversarial prompts