Lessons from Defending Gemini Against Indirect Prompt Injections
Gemini is increasingly used to perform tasks on behalf of users, where
function-calling and tool-use capabilities enable the model to access user
data. Some tools, however, require access to untrusted data introducing risk.
Adversaries can embed malicious instructions in untrusted data which cause the
model to deviate from the user's expectations and mishandle their data or
permissions. In this report, we set out Google DeepMind's approach to
evaluating the adversarial robustness of Gemini models and describe the main
lessons learned from the process. We test how Gemini performs against a
sophisticated adversary through an adversarial evaluation framework, which
deploys a suite of adaptive attack techniques to run continuously against past,
current, and future versions of Gemini. We describe how these ongoing
evaluations directly help make Gemini more resilient against manipulation.
682d7f009a06bc9f9106480b
Chongyang Shi
extracted_pending
2025-05-21T07:21:37.326Z
65ef0b4b325d9aaef87eb33b
/avatars/ec27dba9de7e74c1f6cdcacf5aa25528.svg
C Shi
user
chongyangs
682d7f009a06bc9f9106480c
Sharon Lin
682d7f009a06bc9f9106480d
Shuang Song
682d7f009a06bc9f9106480e
Jamie Hayes
682d7f009a06bc9f9106480f
Ilia Shumailov
admin_assigned
2025-05-21T10:31:09.839Z
6475c2794766357252e69e9f
/avatars/757ed789423113369868e972f21ce559.svg
i
user
iliashum
682d7f009a06bc9f91064810
Itay Yona
admin_assigned
2025-05-21T10:31:16.965Z
62697edaa6a7bba9e46ae4ad
/avatars/3009bf769a09d946447a0e8c833a04f3.svg
Itay Yona
user
tux
682d7f009a06bc9f91064811
Juliette Pluto
admin_assigned
2025-05-21T10:31:22.677Z
657d17d43480ce8aae628d40
/avatars/7f8cd9953f82e5a7ad5db1f82cfdb8f1.svg
juliette pluto
user
julsh
682d7f009a06bc9f91064812
Aneesh Pappu
682d7f009a06bc9f91064813
Christopher A. Choquette-Choo
admin_assigned
2025-05-21T10:31:33.340Z
649c5307e89a7650158f4965
/avatars/3c71b501db27c86ed686ae26ad1391cd.svg
Christopher A. Choquette-Choo
user
cchoquette
682d7f009a06bc9f91064814
Milad Nasr
682d7f009a06bc9f91064815
Chawin Sitawarin
admin_assigned
2025-05-21T10:31:58.593Z
623e3ebef3be5bae969680c0
/avatars/590c264bf9964225bd67f61a3a5d53ea.svg
Chawin Sitawarin
user
chawins
682d7f009a06bc9f91064816
Gena Gibson
682d7f009a06bc9f91064817
Andreas Terzis
682d7f009a06bc9f91064818
John "Four" Flynn
Lessons from Defending Gemini Against Indirect Prompt Injections
Gemini is increasingly used to perform tasks on behalf of users, where
function-calling and tool-use capabilities enable the model to access user
data. Some tools, however, require access to untrusted data introducing risk.
Adversaries can embed malicious instructions in untrusted data which cause the
model to deviate from the user's expectations and mishandle their data or
permissions. In this report, we set out Google DeepMind's approach to
evaluating the adversarial robustness of Gemini models and describe the main
lessons learned from the process. We test how Gemini performs against a
sophisticated adversary through an adversarial evaluation framework, which
deploys a suite of adaptive attack techniques to run continuously against past,
current, and future versions of Gemini. We describe how these ongoing
evaluations directly help make Gemini more resilient against manipulation.
Google DeepMind evaluates the adversarial robustness of Gemini through continuous testing with adaptive attack techniques to enhance its resilience.
adversarial robustness model evaluation adaptive attack techniques