RWKV-7 "Goose" with Expressive Dynamic State Evolution
We present RWKV-7 "Goose", a new sequence modeling architecture, along with
pre-trained language models that establish a new state-of-the-art in downstream
performance at the 3 billion parameter scale on multilingual tasks, and match
current SoTA English language performance despite being trained on dramatically
fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only
constant memory usage and constant inference time per token. RWKV-7 introduces
a newly generalized formulation of the delta rule with vector-valued gating and
in-context learning rates, as well as a relaxed value replacement rule. We show
that RWKV-7 can perform state tracking and recognize all regular languages,
while retaining parallelizability of training. This exceeds the capabilities of
Transformers under standard complexity conjectures, which are limited to
TC^0. To demonstrate RWKV-7's language modeling capability, we also
present an extended open source 3.1 trillion token multilingual corpus, and
train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on
this dataset.
  To foster openness, reproduction, and adoption, we release our models and
dataset component listing at https://huggingface.co/RWKV, and our training and
inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0
License.
67da21ed78c08b432f9fee0c
Bo Peng
claimed_verified
2025-03-19T09:44:45.587Z
62b3d8d651b07307bd12b7f0
https://cdn-avatars.huggingface.co/v1/production/uploads/1655953609090-noauth.jpeg
BlinkDL
user
BlinkDL
67da21ed78c08b432f9fee0d
Ruichong Zhang
claimed_verified
2025-03-19T09:44:54.749Z
6418629fd13ffa408128d7ae
https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png
Zhang Ruichong
user
ZhangRC
67da21ed78c08b432f9fee0e
Daniel Goldstein
claimed_verified
2025-03-19T09:44:57.333Z
647f4bac45baf21ad709fcd0
/avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
Dan Goldstein
user
SmerkyG
67da21ed78c08b432f9fee0f
Eric Alcaide
67da21ed78c08b432f9fee10
Haowen Hou
claimed_verified
2025-03-20T19:17:18.200Z
64414d0a66a62c605c94d14d
https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg
howard-hou
user
howard-hou
67da21ed78c08b432f9fee11
Janna Lu
claimed_verified
2025-03-20T10:46:12.180Z
66b3d98e040c500914ef558f
/avatars/a90f8306dbd7747520ce5b941ee3bbcb.svg
Janna
user
jannalu
67da21ed78c08b432f9fee12
William Merrill
67da21ed78c08b432f9fee13
Guangyu Song
admin_assigned
2025-03-19T09:50:42.957Z
622c062645261ac5cc0bda94
/avatars/ce544b74110f7fe1ad11a3939526f5da.svg
Guangyu Song
user
Guangyu
67da21ed78c08b432f9fee14
Kaifeng Tan
67da21ed78c08b432f9fee15
Saiteja Utpala
claimed_verified
2025-03-19T09:48:50.276Z
638f1fd8c4444c6ca86ff823
/avatars/405807c3868663246cfe371a2034f351.svg
saitejautpala
user
saitejautpala
67da21ed78c08b432f9fee16
Nathan Wilce
admin_assigned
2025-03-19T09:50:59.151Z
63cac1a50932c72f13995d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/63cac1a50932c72f13995d6f/Bd9jEsCL9yWXdyAQCx61J.jpeg
Nathan Wilce
user
m8than
67da21ed78c08b432f9fee17
Johan S. Wind
67da21ed78c08b432f9fee18
Tianyi Wu
67da21ed78c08b432f9fee19
Daniel Wuttke
claimed_verified
2025-03-19T13:15:03.098Z
63f6706a9cbd6730302b80dc
/avatars/dcbbeeeb6b12641b5884df38c5e13766.svg
Dr. Daniel Wuttke
user
hevok
67da21ed78c08b432f9fee1a
Christian Zhou-Zheng
admin_assigned
2025-03-19T09:51:20.835Z
6584f042b378d311dccea501
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vkq1AQhcuZwIjpFDkdgPQ.png
Christian Zhou-Zheng
user
ChristianAzinn
RWKV-7 "Goose" with Expressive Dynamic State Evolution
We present RWKV-7 "Goose", a new sequence modeling architecture, along with
pre-trained language models that establish a new state-of-the-art in downstream
performance at the 3 billion parameter scale on multilingual tasks, and match
current SoTA English language performance despite being trained on dramatically
fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only
constant memory usage and constant inference time per token. RWKV-7 introduces
a newly generalized formulation of the delta rule with vector-valued gating and
in-context learning rates, as well as a relaxed value replacement rule. We show
that RWKV-7 can perform state tracking and recognize all regular languages,
while retaining parallelizability of training. This exceeds the capabilities of
Transformers under standard complexity conjectures, which are limited to
TC^0. To demonstrate RWKV-7's language modeling capability, we also
present an extended open source 3.1 trillion token multilingual corpus, and
train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on
this dataset.
  To foster openness, reproduction, and adoption, we release our models and
dataset component listing at https://huggingface.co/RWKV, and our training and
inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0
License.
RWKV-7 "Goose" achieves state-of-the-art performance in multilingual tasks with optimal memory and inference efficiency, exceeding Transformer capabilities in complexity.
sequence modeling architecture delta rule vector-valued gating in-context learning rates relaxed value replacement rule state tracking parallelizability Transformers $\mathsf{TC}^0$ multilingual corpus