Stealing User Prompts from Mixture of Experts
Mixture-of-Experts (MoE) models improve the efficiency and scalability of
dense language models by routing each token to a small number of experts in
each layer. In this paper, we show how an adversary that can arrange for their
queries to appear in the same batch of examples as a victim's queries can
exploit Expert-Choice-Routing to fully disclose a victim's prompt. We
successfully demonstrate the effectiveness of this attack on a two-layer
Mixtral model, exploiting the tie-handling behavior of the torch.topk CUDA
implementation. Our results show that we can extract the entire prompt using
O({VM}^2) queries (with vocabulary size V and prompt length M) or 100
queries on average per token in the setting we consider. This is the first
attack to exploit architectural flaws for the purpose of extracting user
prompts, introducing a new class of LLM vulnerabilities.
672340ae85f9fe93ec5b20be
Itay Yona
672340ae85f9fe93ec5b20bf
Ilia Shumailov
672340ae85f9fe93ec5b20c0
Jamie Hayes
672340ae85f9fe93ec5b20c1
Nicholas Carlini
Stealing User Prompts from Mixture of Experts
Mixture-of-Experts (MoE) models improve the efficiency and scalability of
dense language models by routing each token to a small number of experts in
each layer. In this paper, we show how an adversary that can arrange for their
queries to appear in the same batch of examples as a victim's queries can
exploit Expert-Choice-Routing to fully disclose a victim's prompt. We
successfully demonstrate the effectiveness of this attack on a two-layer
Mixtral model, exploiting the tie-handling behavior of the torch.topk CUDA
implementation. Our results show that we can extract the entire prompt using
O({VM}^2) queries (with vocabulary size V and prompt length M) or 100
queries on average per token in the setting we consider. This is the first
attack to exploit architectural flaws for the purpose of extracting user
prompts, introducing a new class of LLM vulnerabilities.
Attackers can exploit architectural flaws in Mixture-of-Experts models to extract user prompts by manipulating query batches in a specific manner.
Mixture-of-Experts Expert-Choice-Routing MoE tie-handling behavior torch.topk CUDA