SmallThinker: A Family of Efficient Large Language Models Natively
  Trained for Local Deployment
While frontier large language models (LLMs) continue to push capability
boundaries, their deployment remains confined to GPU-powered cloud
infrastructure. We challenge this paradigm with SmallThinker, a family of LLMs
natively designed - not adapted - for the unique constraints of local devices:
weak computational power, limited memory, and slow storage. Unlike traditional
approaches that mainly compress existing models built for clouds, we architect
SmallThinker from the ground up to thrive within these limitations. Our
innovation lies in a deployment-aware architecture that transforms constraints
into design principles. First, We introduce a two-level sparse structure
combining fine-grained Mixture-of-Experts (MoE) with sparse feed-forward
networks, drastically reducing computational demands without sacrificing model
capacity. Second, to conquer the I/O bottleneck of slow storage, we design a
pre-attention router that enables our co-designed inference engine to prefetch
expert parameters from storage while computing attention, effectively hiding
storage latency that would otherwise cripple on-device inference. Third, for
memory efficiency, we utilize NoPE-RoPE hybrid sparse attention mechanism to
slash KV cache requirements. We release SmallThinker-4B-A0.6B and
SmallThinker-21B-A3B, which achieve state-of-the-art performance scores and
even outperform larger LLMs. Remarkably, our co-designed system mostly
eliminates the need for expensive GPU hardware: with Q4_0 quantization, both
models exceed 20 tokens/s on ordinary consumer CPUs, while consuming only 1GB
and 8GB of memory respectively. SmallThinker is publicly available at
hf.co/PowerInfer/SmallThinker-4BA0.6B-Instruct and
hf.co/PowerInfer/SmallThinker-21BA3B-Instruct.
68883e4aaf872d625c10c63f
Yixin Song
claimed_verified
2025-07-29T07:07:50.992Z
642924f916d4d8293c93af08
https://cdn-avatars.huggingface.co/v1/production/uploads/642924f916d4d8293c93af08/Olp4zFPgGu2E6mFFPXfQO.jpeg
Yixin Song
user
yixinsong
68883e4aaf872d625c10c640
Zhenliang Xue
68883e4aaf872d625c10c641
Dongliang Wei
68883e4aaf872d625c10c642
Feiyang Chen
claimed_verified
2025-07-29T12:51:01.127Z
631df1a5e207d8fe9563c9fb
/avatars/d4f13755944f24a9c7aff25b97565325.svg
Feiyang
user
smallscientist1
68883e4aaf872d625c10c643
Jianxiang Gao
68883e4aaf872d625c10c644
Junchen Liu
claimed_verified
2025-07-29T07:07:44.672Z
666a8735f5d8ceba613494b9
/avatars/3feec27d0b041d28af0fcc3426d1bd82.svg
Junchen
user
Sorrymaker2024
68883e4aaf872d625c10c645
Hangyu Liang
68883e4aaf872d625c10c646
Guangshuo Qin
claimed_verified
2025-07-29T07:07:53.325Z
6709d31dbb2498474f4b11ad
/avatars/e507fb237ed9f0b098e494e75127988e.svg
Guangshuo Qin
user
qsstcl
68883e4aaf872d625c10c647
Chengrong Tian
68883e4aaf872d625c10c648
Bo Wen
68883e4aaf872d625c10c649
Longyu Zhao
68883e4aaf872d625c10c64a
Xinrui Zheng
68883e4aaf872d625c10c64b
Zeyu Mi
claimed_verified
2025-07-30T09:05:17.214Z
6565a74575968d50cc1e51d3
/avatars/cc50e33b4e275aff2c017b7a2d7830e4.svg
Zeyu Mi
user
yzmizeyu
68883e4aaf872d625c10c64c
Haibo Chen
SmallThinker: A Family of Efficient Large Language Models Natively
  Trained for Local Deployment
While frontier large language models (LLMs) continue to push capability
boundaries, their deployment remains confined to GPU-powered cloud
infrastructure. We challenge this paradigm with SmallThinker, a family of LLMs
natively designed - not adapted - for the unique constraints of local devices:
weak computational power, limited memory, and slow storage. Unlike traditional
approaches that mainly compress existing models built for clouds, we architect
SmallThinker from the ground up to thrive within these limitations. Our
innovation lies in a deployment-aware architecture that transforms constraints
into design principles. First, We introduce a two-level sparse structure
combining fine-grained Mixture-of-Experts (MoE) with sparse feed-forward
networks, drastically reducing computational demands without sacrificing model
capacity. Second, to conquer the I/O bottleneck of slow storage, we design a
pre-attention router that enables our co-designed inference engine to prefetch
expert parameters from storage while computing attention, effectively hiding
storage latency that would otherwise cripple on-device inference. Third, for
memory efficiency, we utilize NoPE-RoPE hybrid sparse attention mechanism to
slash KV cache requirements. We release SmallThinker-4B-A0.6B and
SmallThinker-21B-A3B, which achieve state-of-the-art performance scores and
even outperform larger LLMs. Remarkably, our co-designed system mostly
eliminates the need for expensive GPU hardware: with Q4_0 quantization, both
models exceed 20 tokens/s on ordinary consumer CPUs, while consuming only 1GB
and 8GB of memory respectively. SmallThinker is publicly available at
hf.co/PowerInfer/SmallThinker-4BA0.6B-Instruct and
hf.co/PowerInfer/SmallThinker-21BA3B-Instruct.
SmallThinker, a family of LLMs designed for local devices, uses a deployment-aware architecture with sparse structures, pre-attention routing, and hybrid sparse attention to achieve high performance on consumer hardware.
Mixture-of-Experts (MoE) sparse feed-forward networks pre-attention router NoPE-RoPE hybrid sparse attention mechanism Q4_0 quantization