ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language
  Model Born from Transformer
As is known, hybrid quadratic and subquadratic attention models in multi-head
architectures have surpassed both Transformer and Linear RNN models , with
these works primarily focusing on reducing KV complexity and improving
efficiency. For further research on expressiveness, we introduce our series of
models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which
aims to make RNN more expressive and demonstrates state tracking ability beyond
transformers. We work with QRWK 32B based on RWKV-6 architecture, another
approach that reduces the entire knowledge processing time to just 8 hours
using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the
distillation process can utilize any LLM, not just Qwen, and enables knowledge
transfer from larger LLMs to smaller ones with more fewer tokens. We will
explain the detailed process and share our insights on building more powerful
foundation models. Please note that this is an ongoing work that will be
updated continuously. The model checkpoints and source code are available at
https://github.com/yynil/RWKVInside{https://github.com/yynil/RWKVInside},
https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
679843ae7d7b7f8196c61ab7
Lin Yueyu
admin_assigned
2025-01-28T14:10:34.376Z
63a00aa29f1f2baab2034cf8
/avatars/818d104f45cbce2c47d443756fa806c8.svg
Yueyu Lin
user
yueyulin
679843ae7d7b7f8196c61ab8
Li Zhiyuan
679843ae7d7b7f8196c61ab9
Peter Yue
admin_assigned
2025-01-28T14:11:02.358Z
64087a0992033c15073afb8c
/avatars/9c590ab5c6526edce5084169ec7bde2e.svg
peteryue
user
peteryue
679843ae7d7b7f8196c61aba
Liu Xiao
extracted_confirmed
2025-01-28T02:44:02.658Z
6176b32847ee6431f632981e
https://cdn-avatars.huggingface.co/v1/production/uploads/6176b32847ee6431f632981e/02rZ_oLAI0Ll6Y6be7Q9F.jpeg
Xiaoliu.x
user
xiaol
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language
  Model Born from Transformer
As is known, hybrid quadratic and subquadratic attention models in multi-head
architectures have surpassed both Transformer and Linear RNN models , with
these works primarily focusing on reducing KV complexity and improving
efficiency. For further research on expressiveness, we introduce our series of
models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which
aims to make RNN more expressive and demonstrates state tracking ability beyond
transformers. We work with QRWK 32B based on RWKV-6 architecture, another
approach that reduces the entire knowledge processing time to just 8 hours
using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the
distillation process can utilize any LLM, not just Qwen, and enables knowledge
transfer from larger LLMs to smaller ones with more fewer tokens. We will
explain the detailed process and share our insights on building more powerful
foundation models. Please note that this is an ongoing work that will be
updated continuously. The model checkpoints and source code are available at
https://github.com/yynil/RWKVInside{https://github.com/yynil/RWKVInside},
https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
A series of models distilled from Qwen 2.5 using RWKV-7 attention architecture enhance RNN expressiveness and demonstrate improved state tracking compared to transformers, while reducing knowledge processing time and enabling transfer from larger models.
hybrid quadratic subquadratic attention models multi-head architectures Transformer Linear RNN KV complexity RWKV-7 attention RNN expressiveness state tracking QRWK 32B RWKV-6 architecture knowledge processing time GPU knowledge transfer LLM smaller models fewer tokens foundation models