title: FLIRT: Feedback Loop In-context Red Teaming
summary: Warning: this paper contains content that may be inappropriate or offensive.
    As generative models become available for public use in various applications,
  testing and analyzing vulnerabilities of these models has become a priority.
  Here we propose an automatic red teaming framework that evaluates a given model
  and exposes its vulnerabilities against unsafe and inappropriate content
  generation. Our framework uses in-context learning in a feedback loop to red
  team models and trigger them into unsafe content generation. We propose
  different in-context attack strategies to automatically learn effective and
  diverse adversarial prompts for text-to-image models. Our experiments
  demonstrate that compared to baseline approaches, our proposed strategy is
  significantly more effective in exposing vulnerabilities in Stable Diffusion
  (SD) model, even when the latter is enhanced with safety features. Furthermore,
  we demonstrate that the proposed framework is effective for red teaming
  text-to-text models, resulting in significantly higher toxic response
  generation rate compared to previously reported numbers.
paper_authors._id: 64d3165d61b976fdb06d4817
paper_authors.name: Ninareh Mehrabi
paper_authors._id: 64d3165d61b976fdb06d4818
paper_authors.name: Palash Goyal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-09T11:05:59.981Z
paper_authors._id: 61cfa171fab36112e892385e
paper_authors.avatarUrl: /avatars/6110ed2f41733c8ebb17abc7d934479a.svg
paper_authors.fullname: Palash Goyal
paper_authors.type: user
paper_authors.user: palashgo
paper_authors._id: 64d3165d61b976fdb06d4819
paper_authors.name: Christophe Dupuy
paper_authors._id: 64d3165d61b976fdb06d481a
paper_authors.name: Qian Hu
paper_authors._id: 64d3165d61b976fdb06d481b
paper_authors.name: Shalini Ghosh
paper_authors._id: 64d3165d61b976fdb06d481c
paper_authors.name: Richard Zemel
paper_authors._id: 64d3165d61b976fdb06d481d
paper_authors.name: Kai-Wei Chang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-10T01:13:46.007Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 64d3165d61b976fdb06d481e
paper_authors.name: Aram Galstyan
paper_authors._id: 64d3165d61b976fdb06d481f
paper_authors.name: Rahul Gupta
paper_title: FLIRT: Feedback Loop In-context Red Teaming
paper_summary: Warning: this paper contains content that may be inappropriate or offensive.
    As generative models become available for public use in various applications,
  testing and analyzing vulnerabilities of these models has become a priority.
  Here we propose an automatic red teaming framework that evaluates a given model
  and exposes its vulnerabilities against unsafe and inappropriate content
  generation. Our framework uses in-context learning in a feedback loop to red
  team models and trigger them into unsafe content generation. We propose
  different in-context attack strategies to automatically learn effective and
  diverse adversarial prompts for text-to-image models. Our experiments
  demonstrate that compared to baseline approaches, our proposed strategy is
  significantly more effective in exposing vulnerabilities in Stable Diffusion
  (SD) model, even when the latter is enhanced with safety features. Furthermore,
  we demonstrate that the proposed framework is effective for red teaming
  text-to-text models, resulting in significantly higher toxic response
  generation rate compared to previously reported numbers.
paper_ai_summary: An automatic red teaming framework uses in-context learning to identify vulnerabilities in generative models, particularly effective for text-to-image and text-to-text models like Stable Diffusion.
paper_ai_keywords: in-context learning adversarial prompts text-to-image models stable diffusion text-to-text models toxic response generation