title: ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language
    Model Born from Transformer
summary: As is known, hybrid quadratic and subquadratic attention models in multi-head
  architectures have surpassed both Transformer and Linear RNN models , with
  these works primarily focusing on reducing KV complexity and improving
  efficiency. For further research on expressiveness, we introduce our series of
  models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which
  aims to make RNN more expressive and demonstrates state tracking ability beyond
  transformers. We work with QRWK 32B based on RWKV-6 architecture, another
  approach that reduces the entire knowledge processing time to just 8 hours
  using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the
  distillation process can utilize any LLM, not just Qwen, and enables knowledge
  transfer from larger LLMs to smaller ones with more fewer tokens. We will
  explain the detailed process and share our insights on building more powerful
  foundation models. Please note that this is an ongoing work that will be
  updated continuously. The model checkpoints and source code are available at
  https://github.com/yynil/RWKVInside{https://github.com/yynil/RWKVInside},
  https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
paper_authors._id: 679843ae7d7b7f8196c61ab7
paper_authors.name: Lin Yueyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T14:10:34.376Z
paper_authors._id: 63a00aa29f1f2baab2034cf8
paper_authors.avatarUrl: /avatars/818d104f45cbce2c47d443756fa806c8.svg
paper_authors.fullname: Yueyu Lin
paper_authors.type: user
paper_authors.user: yueyulin
paper_authors._id: 679843ae7d7b7f8196c61ab8
paper_authors.name: Li Zhiyuan
paper_authors._id: 679843ae7d7b7f8196c61ab9
paper_authors.name: Peter Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T14:11:02.358Z
paper_authors._id: 64087a0992033c15073afb8c
paper_authors.avatarUrl: /avatars/9c590ab5c6526edce5084169ec7bde2e.svg
paper_authors.fullname: peteryue
paper_authors.type: user
paper_authors.user: peteryue
paper_authors._id: 679843ae7d7b7f8196c61aba
paper_authors.name: Liu Xiao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-28T02:44:02.658Z
paper_authors._id: 6176b32847ee6431f632981e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6176b32847ee6431f632981e/02rZ_oLAI0Ll6Y6be7Q9F.jpeg
paper_authors.fullname: IvanD
paper_authors.type: user
paper_authors.user: xiaol
paper_title: ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language
    Model Born from Transformer
paper_summary: As is known, hybrid quadratic and subquadratic attention models in multi-head
  architectures have surpassed both Transformer and Linear RNN models , with
  these works primarily focusing on reducing KV complexity and improving
  efficiency. For further research on expressiveness, we introduce our series of
  models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which
  aims to make RNN more expressive and demonstrates state tracking ability beyond
  transformers. We work with QRWK 32B based on RWKV-6 architecture, another
  approach that reduces the entire knowledge processing time to just 8 hours
  using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the
  distillation process can utilize any LLM, not just Qwen, and enables knowledge
  transfer from larger LLMs to smaller ones with more fewer tokens. We will
  explain the detailed process and share our insights on building more powerful
  foundation models. Please note that this is an ongoing work that will be
  updated continuously. The model checkpoints and source code are available at
  https://github.com/yynil/RWKVInside{https://github.com/yynil/RWKVInside},
  https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
paper_ai_keywords: hybrid quadratic and subquadratic attention models multi-head architectures Transformer Linear RNN models KV complexity RWKV-7 attention RNN QRWK 32B RWKV-6 architecture state tracking ability knowledge processing time AMD MI300X GPUs LLM knowledge transfer model checkpoints source code