title: V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V
    Platforms
summary: The recent exponential growth of Large Language Models (LLMs) has relied on
  GPU-based systems. However, CPUs are emerging as a flexible and lower-cost
  alternative, especially when targeting inference and reasoning workloads.
  RISC-V is rapidly gaining traction in this area, given its open and
  vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the
  corresponding software ecosystem are not fully mature and streamlined, given
  the requirement of domain-specific tuning. This paper aims at filling this gap,
  focusing on optimizing LLM inference on the Sophon SG2042, the first
  commercially available many-core RISC-V CPU with vector processing
  capabilities.
    On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1
  Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s
  for token generation and 6.54/3.68 token/s for prompt processing, with a speed
  up of up 2.9x/3.0x compared to our baseline.
paper_authors._id: 67e226b3b1acaf8a7680e926
paper_authors.name: Javier J. Poveda Rodrigo
paper_authors._id: 67e226b3b1acaf8a7680e927
paper_authors.name: Mohamed Amine Ahmdi
paper_authors._id: 67e226b3b1acaf8a7680e928
paper_authors.name: Alessio Burrello
paper_authors._id: 67e226b3b1acaf8a7680e929
paper_authors.name: Daniele Jahier Pagliari
paper_authors._id: 67e226b3b1acaf8a7680e92a
paper_authors.name: Luca Benini
paper_title: V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V
    Platforms
paper_summary: The recent exponential growth of Large Language Models (LLMs) has relied on
  GPU-based systems. However, CPUs are emerging as a flexible and lower-cost
  alternative, especially when targeting inference and reasoning workloads.
  RISC-V is rapidly gaining traction in this area, given its open and
  vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the
  corresponding software ecosystem are not fully mature and streamlined, given
  the requirement of domain-specific tuning. This paper aims at filling this gap,
  focusing on optimizing LLM inference on the Sophon SG2042, the first
  commercially available many-core RISC-V CPU with vector processing
  capabilities.
    On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1
  Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s
  for token generation and 6.54/3.68 token/s for prompt processing, with a speed
  up of up 2.9x/3.0x compared to our baseline.
paper_ai_summary: Optimization of large language model inference on RISC-V CPU with vector processing significantly improves speed compared to baseline systems.
paper_ai_keywords: Large Language Models LLMs GPU-based systems CPUs RISC-V vector processing inference prompt processing token generation baseline