title: CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through
    Cryptography Challenges
summary: Large language models (LLMs) have demonstrated remarkable capabilities,
  especially the recent advancements in reasoning, such as o1 and o3, pushing the
  boundaries of AI. Despite these impressive achievements in mathematics and
  coding, the reasoning abilities of LLMs in domains requiring cryptographic
  expertise remain underexplored. In this paper, we introduce CipherBank, a
  comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs
  in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously
  crafted problems, covering 262 unique plaintexts across 5 domains and 14
  subdomains, with a focus on privacy-sensitive and real-world scenarios that
  necessitate encryption. From a cryptographic perspective, CipherBank
  incorporates 3 major categories of encryption methods, spanning 9 distinct
  algorithms, ranging from classical ciphers to custom cryptographic techniques.
  We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and
  cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results
  reveal significant gaps in reasoning abilities not only between general-purpose
  chat LLMs and reasoning-focused LLMs but also in the performance of current
  reasoning-focused models when applied to classical cryptographic decryption
  tasks, highlighting the challenges these models face in understanding and
  manipulating encrypted data. Through detailed analysis and error
  investigations, we provide several key observations that shed light on the
  limitations and potential improvement areas for LLMs in cryptographic
  reasoning. These findings underscore the need for continuous advancements in
  LLM reasoning capabilities.
paper_authors._id: 6810356ab91a093e4f4cc262
paper_authors.name: Yu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-29T08:01:23.352Z
paper_authors._id: 671b852aa4fa4f8f5fb5404c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/671b852aa4fa4f8f5fb5404c/TDLsgP8WgKW-qaA8Ys-iJ.jpeg
paper_authors.fullname: YU LI
paper_authors.type: user
paper_authors.user: yu0226
paper_authors._id: 6810356ab91a093e4f4cc263
paper_authors.name: Qizhi Pei
paper_authors._id: 6810356ab91a093e4f4cc264
paper_authors.name: Mengyuan Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-29T08:01:29.740Z
paper_authors._id: 67ad790c2b28204981be8e24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67ad790c2b28204981be8e24/KstE5e5bUXXIvgPJqMO2B.jpeg
paper_authors.fullname: Mengyuan Sun
paper_authors.type: user
paper_authors.user: blue01223
paper_authors._id: 6810356ab91a093e4f4cc265
paper_authors.name: Honglin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:17:44.229Z
paper_authors._id: 640d99628512ec51d7ef71c7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640d99628512ec51d7ef71c7/fcBkqnxfxuuuZTqfN_BGy.jpeg
paper_authors.fullname: Honglin Lin
paper_authors.type: user
paper_authors.user: LHL3341
paper_authors._id: 6810356ab91a093e4f4cc266
paper_authors.name: Chenlin Ming
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:17:55.935Z
paper_authors._id: 677e133ee86d0754dc7ce296
paper_authors.avatarUrl: /avatars/c16511c1876b50c2d049925c5f320d15.svg
paper_authors.fullname: mingchenlin
paper_authors.type: user
paper_authors.user: mingchenlin2025
paper_authors._id: 6810356ab91a093e4f4cc267
paper_authors.name: Xin Gao
paper_authors._id: 6810356ab91a093e4f4cc268
paper_authors.name: Jiang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:18:06.650Z
paper_authors._id: 64bb936fc6e77d66f46ee99d
paper_authors.avatarUrl: /avatars/5b910a8fced936edcc075314fb353921.svg
paper_authors.fullname: Jiang Wu
paper_authors.type: user
paper_authors.user: JiangWu
paper_authors._id: 6810356ab91a093e4f4cc269
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:18:17.101Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 6810356ab91a093e4f4cc26a
paper_authors.name: Lijun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:18:37.977Z
paper_authors._id: 643e60d96db6ba8c5ee177ad
paper_authors.avatarUrl: /avatars/73ac7740e462ba0b53a2f2480d9f1e3e.svg
paper_authors.fullname: Lijun Wu
paper_authors.type: user
paper_authors.user: apeters
paper_title: CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through
    Cryptography Challenges
paper_summary: Large language models (LLMs) have demonstrated remarkable capabilities,
  especially the recent advancements in reasoning, such as o1 and o3, pushing the
  boundaries of AI. Despite these impressive achievements in mathematics and
  coding, the reasoning abilities of LLMs in domains requiring cryptographic
  expertise remain underexplored. In this paper, we introduce CipherBank, a
  comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs
  in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously
  crafted problems, covering 262 unique plaintexts across 5 domains and 14
  subdomains, with a focus on privacy-sensitive and real-world scenarios that
  necessitate encryption. From a cryptographic perspective, CipherBank
  incorporates 3 major categories of encryption methods, spanning 9 distinct
  algorithms, ranging from classical ciphers to custom cryptographic techniques.
  We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and
  cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results
  reveal significant gaps in reasoning abilities not only between general-purpose
  chat LLMs and reasoning-focused LLMs but also in the performance of current
  reasoning-focused models when applied to classical cryptographic decryption
  tasks, highlighting the challenges these models face in understanding and
  manipulating encrypted data. Through detailed analysis and error
  investigations, we provide several key observations that shed light on the
  limitations and potential improvement areas for LLMs in cryptographic
  reasoning. These findings underscore the need for continuous advancements in
  LLM reasoning capabilities.
paper_ai_summary: CipherBank evaluates the reasoning capabilities of large language models in cryptographic decryption tasks, revealing significant gaps in their performance across classical and custom encryption methods.
paper_ai_keywords: LLMs CipherBank reasoning capabilities cryptographic decryption tasks plaintexts encryption methods classical ciphers custom cryptographic techniques GPT-4o DeepSeek-V3 reasoning-focused models