CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through
  Cryptography Challenges
Large language models (LLMs) have demonstrated remarkable capabilities,
especially the recent advancements in reasoning, such as o1 and o3, pushing the
boundaries of AI. Despite these impressive achievements in mathematics and
coding, the reasoning abilities of LLMs in domains requiring cryptographic
expertise remain underexplored. In this paper, we introduce CipherBank, a
comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs
in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously
crafted problems, covering 262 unique plaintexts across 5 domains and 14
subdomains, with a focus on privacy-sensitive and real-world scenarios that
necessitate encryption. From a cryptographic perspective, CipherBank
incorporates 3 major categories of encryption methods, spanning 9 distinct
algorithms, ranging from classical ciphers to custom cryptographic techniques.
We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and
cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results
reveal significant gaps in reasoning abilities not only between general-purpose
chat LLMs and reasoning-focused LLMs but also in the performance of current
reasoning-focused models when applied to classical cryptographic decryption
tasks, highlighting the challenges these models face in understanding and
manipulating encrypted data. Through detailed analysis and error
investigations, we provide several key observations that shed light on the
limitations and potential improvement areas for LLMs in cryptographic
reasoning. These findings underscore the need for continuous advancements in
LLM reasoning capabilities.
6810356ab91a093e4f4cc262
Yu Li
claimed_verified
2025-04-29T08:01:23.352Z
671b852aa4fa4f8f5fb5404c
https://cdn-avatars.huggingface.co/v1/production/uploads/671b852aa4fa4f8f5fb5404c/TDLsgP8WgKW-qaA8Ys-iJ.jpeg
YU LI
user
yu0226
6810356ab91a093e4f4cc263
Qizhi Pei
6810356ab91a093e4f4cc264
Mengyuan Sun
claimed_verified
2025-04-29T08:01:29.740Z
67ad790c2b28204981be8e24
https://cdn-avatars.huggingface.co/v1/production/uploads/67ad790c2b28204981be8e24/KstE5e5bUXXIvgPJqMO2B.jpeg
Mengyuan Sun
user
blue01223
6810356ab91a093e4f4cc265
Honglin Lin
admin_assigned
2025-04-29T12:17:44.229Z
640d99628512ec51d7ef71c7
https://cdn-avatars.huggingface.co/v1/production/uploads/640d99628512ec51d7ef71c7/fcBkqnxfxuuuZTqfN_BGy.jpeg
Honglin Lin
user
LHL3341
6810356ab91a093e4f4cc266
Chenlin Ming
admin_assigned
2025-04-29T12:17:55.935Z
677e133ee86d0754dc7ce296
/avatars/c16511c1876b50c2d049925c5f320d15.svg
mingchenlin
user
mingchenlin2025
6810356ab91a093e4f4cc267
Xin Gao
6810356ab91a093e4f4cc268
Jiang Wu
admin_assigned
2025-04-29T12:18:06.650Z
64bb936fc6e77d66f46ee99d
/avatars/5b910a8fced936edcc075314fb353921.svg
Jiang Wu
user
JiangWu
6810356ab91a093e4f4cc269
Conghui He
admin_assigned
2025-04-29T12:18:17.101Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
6810356ab91a093e4f4cc26a
Lijun Wu
admin_assigned
2025-04-29T12:18:37.977Z
643e60d96db6ba8c5ee177ad
/avatars/73ac7740e462ba0b53a2f2480d9f1e3e.svg
Lijun Wu
user
apeters
CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through
  Cryptography Challenges
Large language models (LLMs) have demonstrated remarkable capabilities,
especially the recent advancements in reasoning, such as o1 and o3, pushing the
boundaries of AI. Despite these impressive achievements in mathematics and
coding, the reasoning abilities of LLMs in domains requiring cryptographic
expertise remain underexplored. In this paper, we introduce CipherBank, a
comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs
in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously
crafted problems, covering 262 unique plaintexts across 5 domains and 14
subdomains, with a focus on privacy-sensitive and real-world scenarios that
necessitate encryption. From a cryptographic perspective, CipherBank
incorporates 3 major categories of encryption methods, spanning 9 distinct
algorithms, ranging from classical ciphers to custom cryptographic techniques.
We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and
cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results
reveal significant gaps in reasoning abilities not only between general-purpose
chat LLMs and reasoning-focused LLMs but also in the performance of current
reasoning-focused models when applied to classical cryptographic decryption
tasks, highlighting the challenges these models face in understanding and
manipulating encrypted data. Through detailed analysis and error
investigations, we provide several key observations that shed light on the
limitations and potential improvement areas for LLMs in cryptographic
reasoning. These findings underscore the need for continuous advancements in
LLM reasoning capabilities.
CipherBank evaluates the reasoning capabilities of large language models in cryptographic decryption tasks, revealing significant gaps in their performance across classical and custom encryption methods.
LLMs CipherBank reasoning capabilities cryptographic decryption tasks plaintexts encryption methods classical ciphers custom cryptographic techniques GPT-4o DeepSeek-V3 reasoning-focused models