V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V
  Platforms
The recent exponential growth of Large Language Models (LLMs) has relied on
GPU-based systems. However, CPUs are emerging as a flexible and lower-cost
alternative, especially when targeting inference and reasoning workloads.
RISC-V is rapidly gaining traction in this area, given its open and
vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the
corresponding software ecosystem are not fully mature and streamlined, given
the requirement of domain-specific tuning. This paper aims at filling this gap,
focusing on optimizing LLM inference on the Sophon SG2042, the first
commercially available many-core RISC-V CPU with vector processing
capabilities.
  On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1
Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s
for token generation and 6.54/3.68 token/s for prompt processing, with a speed
up of up 2.9x/3.0x compared to our baseline.
67e226b3b1acaf8a7680e926
Javier J. Poveda Rodrigo
67e226b3b1acaf8a7680e927
Mohamed Amine Ahmdi
67e226b3b1acaf8a7680e928
Alessio Burrello
67e226b3b1acaf8a7680e929
Daniele Jahier Pagliari
67e226b3b1acaf8a7680e92a
Luca Benini
V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V
  Platforms
The recent exponential growth of Large Language Models (LLMs) has relied on
GPU-based systems. However, CPUs are emerging as a flexible and lower-cost
alternative, especially when targeting inference and reasoning workloads.
RISC-V is rapidly gaining traction in this area, given its open and
vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the
corresponding software ecosystem are not fully mature and streamlined, given
the requirement of domain-specific tuning. This paper aims at filling this gap,
focusing on optimizing LLM inference on the Sophon SG2042, the first
commercially available many-core RISC-V CPU with vector processing
capabilities.
  On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1
Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s
for token generation and 6.54/3.68 token/s for prompt processing, with a speed
up of up 2.9x/3.0x compared to our baseline.
Optimization of large language model inference on RISC-V CPU with vector processing significantly improves speed compared to baseline systems.
Large Language Models LLMs GPU-based systems CPUs RISC-V vector processing inference prompt processing token generation baseline