Prompt injection: what’s the worst that can happen?
The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code.<p>Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, <i>however</i> if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can&#x27;t possibly be expressed (and thus also overriden) by some text content.<p>Instead of having some words specifying &quot;assistant do this, this is a prompt&quot; you can use explicit special tokens (something which can&#x27;t result from <i>any</i> user-provided data and has to be placed there by system code) as separators or literally just add a single one-bit neuron to the vectors of every token that specifies &quot;this is a prompt&quot; and train your reinforcement learning layer to ignore any instructions without that &quot;privilege&quot; bit set. Or add an explicit one-bit neuron to each token which states &quot;did this text come in from an external source like webpage or email or API call&quot;.<p>[edit] this &#x27;just&#x27; does gloss over technical issues, such as handling that during pre-training, the need for masking something, as for performance reasons we do want the vectors to be multiples of specific numbers and not just an odd number, etc - but I think the concept is simple enough that it can&#x27;t be an obstacle but just a reasonable engineering task.