[WIP] Jailbreak Paradox: The Achilles' Heel of LLMs
<|reference_start|>[WIP] Jailbreak Paradox: The Achilles' Heel of LLMs: We introduce two paradoxes concerning jailbreak of foundation models: First, it is impossible to construct a perfect jailbreak classifier, and second, a weaker model cannot consistently detect whether a stronger (in a pareto-dominant sense) model is jailbroken or not. We provide formal proofs for these paradoxes and a short case study on Llama and GPT4-o to demonstrate this. We discuss broader theoretical and practical repercussions of these results.<|reference_end|>
@article{rao2024[wip],
    title={[WIP] Jailbreak Paradox: The Achilles' Heel of LLMs},
    author={Abhinav Rao, Monojit Choudhury, Somak Aditya},
    journal={arXiv preprint arXiv:2406.12702},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12702},
    primaryClass={cs.CL}
}