Better Language Model Inversion by Compactly Representing Next-Token
  Distributions
Language model inversion seeks to recover hidden prompts using only language
model outputs. This capability has implications for security and accountability
in language model deployments, such as leaking private information from an
API-protected language model's system message. We propose a new method --
prompt inversion from logprob sequences (PILS) -- that recovers hidden prompts
by gleaning clues from the model's next-token probabilities over the course of
multiple generation steps. Our method is enabled by a key insight: The
vector-valued outputs of a language model occupy a low-dimensional subspace.
This enables us to losslessly compress the full next-token probability
distribution over multiple generation steps using a linear map, allowing more
output information to be used for inversion. Our approach yields massive gains
over previous state-of-the-art methods for recovering hidden prompts, achieving
2--3.5 times higher exact recovery rates across test sets, in one case
increasing the recovery rate from 17% to 60%. Our method also exhibits
surprisingly good generalization behavior; for instance, an inverter trained on
16 generations steps gets 5--27 points higher prompt recovery when we increase
the number of steps to 32 at test time. Furthermore, we demonstrate strong
performance of our method on the more challenging task of recovering hidden
system messages. We also analyze the role of verbatim repetition in prompt
recovery and propose a new method for cross-family model transfer for
logit-based inverters. Our findings show that next-token probabilities are a
considerably more vulnerable attack surface for inversion attacks than
previously known.
685968490e4ad7e2197583a6
Murtaza Nazir
685968490e4ad7e2197583a7
Matthew Finlayson
685968490e4ad7e2197583a8
John X. Morris
685968490e4ad7e2197583a9
Xiang Ren
685968490e4ad7e2197583aa
Swabha Swayamdipta
claimed_verified
2025-07-31T08:16:19.826Z
688b03af9c6feb94ac299fc9
/avatars/bf94918c198fe866a7d4056ea887108f.svg
Swabha Swayamdipta
user
swabhs
Better Language Model Inversion by Compactly Representing Next-Token
  Distributions
Language model inversion seeks to recover hidden prompts using only language
model outputs. This capability has implications for security and accountability
in language model deployments, such as leaking private information from an
API-protected language model's system message. We propose a new method --
prompt inversion from logprob sequences (PILS) -- that recovers hidden prompts
by gleaning clues from the model's next-token probabilities over the course of
multiple generation steps. Our method is enabled by a key insight: The
vector-valued outputs of a language model occupy a low-dimensional subspace.
This enables us to losslessly compress the full next-token probability
distribution over multiple generation steps using a linear map, allowing more
output information to be used for inversion. Our approach yields massive gains
over previous state-of-the-art methods for recovering hidden prompts, achieving
2--3.5 times higher exact recovery rates across test sets, in one case
increasing the recovery rate from 17% to 60%. Our method also exhibits
surprisingly good generalization behavior; for instance, an inverter trained on
16 generations steps gets 5--27 points higher prompt recovery when we increase
the number of steps to 32 at test time. Furthermore, we demonstrate strong
performance of our method on the more challenging task of recovering hidden
system messages. We also analyze the role of verbatim repetition in prompt
recovery and propose a new method for cross-family model transfer for
logit-based inverters. Our findings show that next-token probabilities are a
considerably more vulnerable attack surface for inversion attacks than
previously known.
A new method called Prompt Inversion from Logprob Sequences (PILS) recovers hidden prompts in language models by analyzing the low-dimensional subspace of the model's next-token probabilities, achieving higher recovery rates and better generalization than previous methods.
Prompt Inversion from Logprob Sequences (PILS) next-token probabilities low-dimensional subspace logprob sequences prompt recovery system message recovery verbatim repetition cross-family model transfer logit-based inverters