Knowledge-to-Jailbreak: One Knowledge Point Worth One Attack
<|reference_start|>Knowledge-to-Jailbreak: One Knowledge Point Worth One Attack: Large language models (LLMs) have been increasingly applied to various domains, which triggers increasing concerns about LLMs' safety on specialized domains, e.g. medicine. However, testing the domain-specific safety of LLMs is challenging due to the lack of domain knowledge-driven attacks in existing benchmarks. To bridge this gap, we propose a new task, knowledge-to-jailbreak, which aims to generate jailbreaks from domain knowledge to evaluate the safety of LLMs when applied to those domains. We collect a large-scale dataset with 12,974 knowledge-jailbreak pairs and fine-tune a large language model as jailbreak-generator, to produce domain knowledge-specific jailbreaks. Experiments on 13 domains and 8 target LLMs demonstrate the effectiveness of jailbreak-generator in generating jailbreaks that are both relevant to the given knowledge and harmful to the target LLMs. We also apply our method to an out-of-domain knowledge base, showing that jailbreak-generator can generate jailbreaks that are comparable in harmfulness to those crafted by human experts. Data and code: https://github.com/THU-KEG/Knowledge-to-Jailbreak/.<|reference_end|>
@article{tu2024knowledge-to-jailbreak:,
    title={Knowledge-to-Jailbreak: One Knowledge Point Worth One Attack},
    author={Shangqing Tu, Zhuoran Pan, Wenxuan Wang, Zhexin Zhang, Yuliang Sun,
  Jifan Yu, Hongning Wang, Lei Hou, Juanzi Li},
    journal={arXiv preprint arXiv:2406.11682},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11682},
    primaryClass={cs.CL cs.AI cs.CR}
}