AI-powered Bing Chat spills its secrets via prompt injection attack
It's very interesting that AppSec may now begin to include "prompt injection" attacks as something of relevance.<p>Specifically with libraries like LangChain[0] that allow for you to perform complex actions ("What's the weather?" -> makes HTTP request to fetch weather) then we end up in a world where injection attacks can have side effects with security implications.<p>I've been thinking about what security might look like for a post-ChatGPT world and how I'd attempt to defend against it. I'd probably start by building a database of attack prompts, kind of like this[1] fuzz list but for AI, then I'd train a second neural net that acts like an adversarial neural network[2] to try to exploit the system based on those payloads. The end result would sort of like SQLMap[3] but for AI systems where it can automatically "leak" hidden prompts and potentially find "bypasses" to escape the sandbox.<p>Has anybody else spent any time thinking about how to defend systems against prompt injection attacks that have possible side effects (like making an HTTP request)?<p>0: <a href="https://langchain.readthedocs.io/en/latest/modules/agents/examples/search_tools.html" rel="nofollow">https://langchain.readthedocs.io/en/latest/modules/agents/ex...</a><p>1: <a href="https://github.com/1N3/IntruderPayloads">https://github.com/1N3/IntruderPayloads</a><p>2: <a href="https://en.wikipedia.org/wiki/Generative_adversarial_network" rel="nofollow">https://en.wikipedia.org/wiki/Generative_adversarial_network</a><p>3: <a href="https://sqlmap.org/" rel="nofollow">https://sqlmap.org/</a>