FLIRT: Feedback Loop In-context Red Teaming
Warning: this paper contains content that may be inappropriate or offensive.
  As generative models become available for public use in various applications,
testing and analyzing vulnerabilities of these models has become a priority.
Here we propose an automatic red teaming framework that evaluates a given model
and exposes its vulnerabilities against unsafe and inappropriate content
generation. Our framework uses in-context learning in a feedback loop to red
team models and trigger them into unsafe content generation. We propose
different in-context attack strategies to automatically learn effective and
diverse adversarial prompts for text-to-image models. Our experiments
demonstrate that compared to baseline approaches, our proposed strategy is
significantly more effective in exposing vulnerabilities in Stable Diffusion
(SD) model, even when the latter is enhanced with safety features. Furthermore,
we demonstrate that the proposed framework is effective for red teaming
text-to-text models, resulting in significantly higher toxic response
generation rate compared to previously reported numbers.
64d3165d61b976fdb06d4817
Ninareh Mehrabi
64d3165d61b976fdb06d4818
Palash Goyal
admin_assigned
2023-08-09T11:05:59.981Z
61cfa171fab36112e892385e
/avatars/6110ed2f41733c8ebb17abc7d934479a.svg
Palash Goyal
user
palashgo
64d3165d61b976fdb06d4819
Christophe Dupuy
64d3165d61b976fdb06d481a
Qian Hu
64d3165d61b976fdb06d481b
Shalini Ghosh
64d3165d61b976fdb06d481c
Richard Zemel
64d3165d61b976fdb06d481d
Kai-Wei Chang
claimed_verified
2023-08-10T01:13:46.007Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
64d3165d61b976fdb06d481e
Aram Galstyan
64d3165d61b976fdb06d481f
Rahul Gupta
FLIRT: Feedback Loop In-context Red Teaming
Warning: this paper contains content that may be inappropriate or offensive.
  As generative models become available for public use in various applications,
testing and analyzing vulnerabilities of these models has become a priority.
Here we propose an automatic red teaming framework that evaluates a given model
and exposes its vulnerabilities against unsafe and inappropriate content
generation. Our framework uses in-context learning in a feedback loop to red
team models and trigger them into unsafe content generation. We propose
different in-context attack strategies to automatically learn effective and
diverse adversarial prompts for text-to-image models. Our experiments
demonstrate that compared to baseline approaches, our proposed strategy is
significantly more effective in exposing vulnerabilities in Stable Diffusion
(SD) model, even when the latter is enhanced with safety features. Furthermore,
we demonstrate that the proposed framework is effective for red teaming
text-to-text models, resulting in significantly higher toxic response
generation rate compared to previously reported numbers.
An automatic red teaming framework uses in-context learning to identify vulnerabilities in generative models, particularly effective for text-to-image and text-to-text models like Stable Diffusion.
in-context learning adversarial prompts text-to-image models stable diffusion text-to-text models toxic response generation