title: RWKV-7 "Goose" with Expressive Dynamic State Evolution
summary: We present RWKV-7 "Goose", a new sequence modeling architecture, along with
  pre-trained language models that establish a new state-of-the-art in downstream
  performance at the 3 billion parameter scale on multilingual tasks, and match
  current SoTA English language performance despite being trained on dramatically
  fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only
  constant memory usage and constant inference time per token. RWKV-7 introduces
  a newly generalized formulation of the delta rule with vector-valued gating and
  in-context learning rates, as well as a relaxed value replacement rule. We show
  that RWKV-7 can perform state tracking and recognize all regular languages,
  while retaining parallelizability of training. This exceeds the capabilities of
  Transformers under standard complexity conjectures, which are limited to
  TC^0. To demonstrate RWKV-7's language modeling capability, we also
  present an extended open source 3.1 trillion token multilingual corpus, and
  train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on
  this dataset.
    To foster openness, reproduction, and adoption, we release our models and
  dataset component listing at https://huggingface.co/RWKV, and our training and
  inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0
  License.
paper_authors._id: 67da21ed78c08b432f9fee0c
paper_authors.name: Bo Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:44:45.587Z
paper_authors._id: 62b3d8d651b07307bd12b7f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655953609090-noauth.jpeg
paper_authors.fullname: BlinkDL
paper_authors.type: user
paper_authors.user: BlinkDL
paper_authors._id: 67da21ed78c08b432f9fee0d
paper_authors.name: Ruichong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:44:54.749Z
paper_authors._id: 6418629fd13ffa408128d7ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png
paper_authors.fullname: Zhang Ruichong
paper_authors.type: user
paper_authors.user: ZhangRC
paper_authors._id: 67da21ed78c08b432f9fee0e
paper_authors.name: Daniel Goldstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:44:57.333Z
paper_authors._id: 647f4bac45baf21ad709fcd0
paper_authors.avatarUrl: /avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
paper_authors.fullname: Dan Goldstein
paper_authors.type: user
paper_authors.user: SmerkyG
paper_authors._id: 67da21ed78c08b432f9fee0f
paper_authors.name: Eric Alcaide
paper_authors._id: 67da21ed78c08b432f9fee10
paper_authors.name: Haowen Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T19:17:18.200Z
paper_authors._id: 64414d0a66a62c605c94d14d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg
paper_authors.fullname: howard-hou
paper_authors.type: user
paper_authors.user: howard-hou
paper_authors._id: 67da21ed78c08b432f9fee11
paper_authors.name: Janna Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:12.180Z
paper_authors._id: 66b3d98e040c500914ef558f
paper_authors.avatarUrl: /avatars/a90f8306dbd7747520ce5b941ee3bbcb.svg
paper_authors.fullname: Janna
paper_authors.type: user
paper_authors.user: jannalu
paper_authors._id: 67da21ed78c08b432f9fee12
paper_authors.name: William Merrill
paper_authors._id: 67da21ed78c08b432f9fee13
paper_authors.name: Guangyu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T09:50:42.957Z
paper_authors._id: 622c062645261ac5cc0bda94
paper_authors.avatarUrl: /avatars/ce544b74110f7fe1ad11a3939526f5da.svg
paper_authors.fullname: Guangyu Song
paper_authors.type: user
paper_authors.user: Guangyu
paper_authors._id: 67da21ed78c08b432f9fee14
paper_authors.name: Kaifeng Tan
paper_authors._id: 67da21ed78c08b432f9fee15
paper_authors.name: Saiteja Utpala
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:48:50.276Z
paper_authors._id: 638f1fd8c4444c6ca86ff823
paper_authors.avatarUrl: /avatars/405807c3868663246cfe371a2034f351.svg
paper_authors.fullname: saitejautpala
paper_authors.type: user
paper_authors.user: saitejautpala
paper_authors._id: 67da21ed78c08b432f9fee16
paper_authors.name: Nathan Wilce
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T09:50:59.151Z
paper_authors._id: 63cac1a50932c72f13995d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63cac1a50932c72f13995d6f/Bd9jEsCL9yWXdyAQCx61J.jpeg
paper_authors.fullname: Nathan Wilce
paper_authors.type: user
paper_authors.user: m8than
paper_authors._id: 67da21ed78c08b432f9fee17
paper_authors.name: Johan S. Wind
paper_authors._id: 67da21ed78c08b432f9fee18
paper_authors.name: Tianyi Wu
paper_authors._id: 67da21ed78c08b432f9fee19
paper_authors.name: Daniel Wuttke
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T13:15:03.098Z
paper_authors._id: 63f6706a9cbd6730302b80dc
paper_authors.avatarUrl: /avatars/dcbbeeeb6b12641b5884df38c5e13766.svg
paper_authors.fullname: Dr. Daniel Wuttke
paper_authors.type: user
paper_authors.user: hevok
paper_authors._id: 67da21ed78c08b432f9fee1a
paper_authors.name: Christian Zhou-Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T09:51:20.835Z
paper_authors._id: 6584f042b378d311dccea501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vkq1AQhcuZwIjpFDkdgPQ.png
paper_authors.fullname: Christian Zhou-Zheng
paper_authors.type: user
paper_authors.user: ChristianAzinn
paper_title: RWKV-7 "Goose" with Expressive Dynamic State Evolution
paper_summary: We present RWKV-7 "Goose", a new sequence modeling architecture, along with
  pre-trained language models that establish a new state-of-the-art in downstream
  performance at the 3 billion parameter scale on multilingual tasks, and match
  current SoTA English language performance despite being trained on dramatically
  fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only
  constant memory usage and constant inference time per token. RWKV-7 introduces
  a newly generalized formulation of the delta rule with vector-valued gating and
  in-context learning rates, as well as a relaxed value replacement rule. We show
  that RWKV-7 can perform state tracking and recognize all regular languages,
  while retaining parallelizability of training. This exceeds the capabilities of
  Transformers under standard complexity conjectures, which are limited to
  TC^0. To demonstrate RWKV-7's language modeling capability, we also
  present an extended open source 3.1 trillion token multilingual corpus, and
  train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on
  this dataset.
    To foster openness, reproduction, and adoption, we release our models and
  dataset component listing at https://huggingface.co/RWKV, and our training and
  inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0
  License.
paper_ai_summary: RWKV-7 "Goose" achieves state-of-the-art performance in multilingual tasks with optimal memory and inference efficiency, exceeding Transformer capabilities in complexity.
paper_ai_keywords: sequence modeling architecture delta rule vector-valued gating in-context learning rates relaxed value replacement rule state tracking parallelizability Transformers $\mathsf{TC}^0$ multilingual corpus