We present a dataset of over 100,000 prompt injection attacks and 30,000 anti-injection "defense" prompts created by players of an online game, Tensor Trust. To the best of our knowledge, it is the largest dataset of human-generated adversarial examples for instruction-following LLMs. Using the Tensor Trust dataset, we create benchmarks for resistance to two types of prompt injection (which we refer to as prompt extraction and prompt hijacking) as well as a benchmark for detecting when an LLM has leaked sensitive information from the prompt. We also show that many attacks in our dataset have an intuitive structure that sheds light on the weaknesses of these models. The full Tensor Trust dataset and source code are available at `[URL removed for review]`.
Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game
Sam Toyer Olivia Watkins Ethan Adrian Mendes Justin Svegliato Luke Bailey Tiffany Wang Isaac Ong Karim Elmaaroufi Pieter Abbeel Trevor Darrell Alan Ritter Stuart Russell