title: Efficient Detection of Toxic Prompts in Large Language Models
summary: Large language models (LLMs) like ChatGPT and Gemini have significantly
  advanced natural language processing, enabling various applications such as
  chatbots and automated content generation. However, these models can be
  exploited by malicious individuals who craft toxic prompts to elicit harmful or
  unethical responses. These individuals often employ jailbreaking techniques to
  bypass safety mechanisms, highlighting the need for robust toxic prompt
  detection methods. Existing detection techniques, both blackbox and whitebox,
  face challenges related to the diversity of toxic prompts, scalability, and
  computational efficiency. In response, we propose ToxicDetector, a lightweight
  greybox method designed to efficiently detect toxic prompts in LLMs.
  ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding
  vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP)
  classifier for prompt classification. Our evaluation on various versions of the
  LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector
  achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%,
  outperforming state-of-the-art methods. Additionally, ToxicDetector's
  processing time of 0.0780 seconds per prompt makes it highly suitable for
  real-time applications. ToxicDetector achieves high accuracy, efficiency, and
  scalability, making it a practical method for toxic prompt detection in LLMs.
paper_authors._id: 66c6b1d759ff3e4811a05026
paper_authors.name: Yi Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:28.173Z
paper_authors._id: 63c6b13950cc81901da4da6c
paper_authors.avatarUrl: /avatars/23f2016e7c7ad902c5750e2ddc0b6358.svg
paper_authors.fullname: YI LIU
paper_authors.type: user
paper_authors.user: sumleo
paper_authors._id: 66c6b1d759ff3e4811a05027
paper_authors.name: Junzhe Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:31:40.415Z
paper_authors._id: 65e6c52354d7e3a708444bcc
paper_authors.avatarUrl: /avatars/b1f6dc0e3c4b4b103e60aa306c8b7c11.svg
paper_authors.fullname: Junzhe Yu
paper_authors.type: user
paper_authors.user: kirkyDig
paper_authors._id: 66c6b1d759ff3e4811a05028
paper_authors.name: Huijia Sun
paper_authors._id: 66c6b1d759ff3e4811a05029
paper_authors.name: Ling Shi
paper_authors._id: 66c6b1d759ff3e4811a0502a
paper_authors.name: Gelei Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:32:12.147Z
paper_authors._id: 64ae49347a9df1bd416f7e21
paper_authors.avatarUrl: /avatars/f8226d99e8f272906e8934c3487ca6e2.svg
paper_authors.fullname: Deng
paper_authors.type: user
paper_authors.user: Gelei
paper_authors._id: 66c6b1d759ff3e4811a0502b
paper_authors.name: Yuqi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:32:18.190Z
paper_authors._id: 63914edd2e1b430e96cf4d59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670467355858-63914edd2e1b430e96cf4d59.jpeg
paper_authors.fullname: Yuqi Chen
paper_authors.type: user
paper_authors.user: YuqiChen
paper_authors._id: 66c6b1d759ff3e4811a0502c
paper_authors.name: Yang Liu
paper_title: Efficient Detection of Toxic Prompts in Large Language Models
paper_summary: Large language models (LLMs) like ChatGPT and Gemini have significantly
  advanced natural language processing, enabling various applications such as
  chatbots and automated content generation. However, these models can be
  exploited by malicious individuals who craft toxic prompts to elicit harmful or
  unethical responses. These individuals often employ jailbreaking techniques to
  bypass safety mechanisms, highlighting the need for robust toxic prompt
  detection methods. Existing detection techniques, both blackbox and whitebox,
  face challenges related to the diversity of toxic prompts, scalability, and
  computational efficiency. In response, we propose ToxicDetector, a lightweight
  greybox method designed to efficiently detect toxic prompts in LLMs.
  ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding
  vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP)
  classifier for prompt classification. Our evaluation on various versions of the
  LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector
  achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%,
  outperforming state-of-the-art methods. Additionally, ToxicDetector's
  processing time of 0.0780 seconds per prompt makes it highly suitable for
  real-time applications. ToxicDetector achieves high accuracy, efficiency, and
  scalability, making it a practical method for toxic prompt detection in LLMs.
paper_ai_keywords: large language models (LLMs) ChatGPT Gemini natural language processing chatbots automated content generation toxic prompts jailbreaking techniques safety mechanisms blackbox detection whitebox detection greybox method Multi-Layer Perceptron (MLP) embedding vectors feature vectors prompt classification LLama models Gemma-2 false positive rate