False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize

🚨False Sense of Security: Our new paper identifies a critical limitation in representation probing-based malicious input detection—purported "high detection accuracy" may confer a false sense of security:

A core finding: Representation-based probing classifiers achieve ≥98% accuracy on in-distribution safety tests, but exhibit significant performance degradation (15%–99% drop) on out-of-distribution data, indicating failure to learn genuine harmful semantics.

We further conducted comparative experiments: First, even simple n-gram Naive Bayes models achieved comparable performance to sophisticated probing tools. This suggests probing classifiers may surface-level patterns over semantic harm detection.

Further validation: When we retained structural features of malicious datasets but replaced harmful content (e.g., "bomb fabrication") with benign alternatives (e.g., "bread making"), probing accuracy plummeted by 60–90%, confirming structural bias over harm recognition.

Analysis of learned patterns reveals two key cues: 1) Instructional linguistic formats (e.g., "how to…") and 2) spurious "malicious-associated" trigger words. Structural paraphrasing restores accuracy, while adding triggers to benign text inflates false positives.

This work raises broader questions: If probing relies on surface cues, do existing probing-based insights (e.g., on truthfulness or hallucinations) lack generalizability? Reevaluation of prior conclusions may be necessary.