Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon
the RWKV (RWKV-4) architecture. Our architectural design advancements include
multi-headed matrix-valued states and a dynamic recurrence mechanism that
improve expressivity while maintaining the inference efficiency characteristics
of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a
fast tokenizer based on greedy matching for enhanced multilinguality. We
trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two
Finch models with 1.6 and 3.1 billion parameters and find that they achieve
competitive performance across a wide variety of benchmarks. We release all our
models on HuggingFace under the Apache 2.0 license. Models at:
https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM
Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code
at: https://github.com/RWKV/RWKV-infctx-trainer
6615f99924c94fd8a22311c1
Bo Peng
claimed_verified
2025-03-19T09:49:08.497Z
62b3d8d651b07307bd12b7f0
https://cdn-avatars.huggingface.co/v1/production/uploads/1655953609090-noauth.jpeg
BlinkDL
user
BlinkDL
6615f99924c94fd8a22311c2
Daniel Goldstein
claimed_verified
2025-03-19T09:49:10.506Z
647f4bac45baf21ad709fcd0
/avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
Dan Goldstein
user
SmerkyG
6615f99924c94fd8a22311c3
Quentin Anthony
claimed_verified
2024-06-20T07:34:51.007Z
63b9a261efe99543b34e9579
https://cdn-avatars.huggingface.co/v1/production/uploads/63b9a261efe99543b34e9579/aZMhhnurORuPU_WryDUiu.jpeg
Quentin Anthony
user
qanthony
6615f99924c94fd8a22311c4
Alon Albalak
claimed_verified
2024-05-15T10:43:23.918Z
611a7ec4289467cafea62d13
https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
Alon Albalak
user
alon-albalak
6615f99924c94fd8a22311c5
Eric Alcaide
claimed_verified
2024-04-14T12:54:30.685Z
646c8a411ee398a4e940cab1
https://cdn-avatars.huggingface.co/v1/production/uploads/646c8a411ee398a4e940cab1/5W3dGj8eQqQrxeGPQO1n2.png
hypnopump
user
hypnopump
6615f99924c94fd8a22311c6
Stella Biderman
claimed_verified
2025-06-07T05:51:30.096Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
6615f99924c94fd8a22311c7
Eugene Cheah
6615f99924c94fd8a22311c8
Teddy Ferdinan
6615f99924c94fd8a22311c9
Haowen Hou
claimed_verified
2025-03-20T19:17:49.238Z
64414d0a66a62c605c94d14d
https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg
howard-hou
user
howard-hou
6615f99924c94fd8a22311ca
Przemysław Kazienko
6615f99924c94fd8a22311cb
Kranthi Kiran GV
6615f99924c94fd8a22311cc
Jan Kocoń
6615f99924c94fd8a22311cd
Bartłomiej Koptyra
6615f99924c94fd8a22311ce
Satyapriya Krishna
claimed_verified
2024-04-23T07:15:39.514Z
6186fef1b1085ab638324e7f
https://cdn-avatars.huggingface.co/v1/production/uploads/6186fef1b1085ab638324e7f/BL6_WJCkxB-BatBUBilT8.jpeg
Satya
user
skrishna
6615f99924c94fd8a22311cf
Ronald McClelland Jr.
6615f99924c94fd8a22311d0
Niklas Muennighoff
6615f99924c94fd8a22311d1
Fares Obeid
claimed_verified
2024-04-10T10:58:45.716Z
643c7c0586ab6dbe34f1eae5
/avatars/aba29ed4a6092658900cf16f32c90f02.svg
Fares Obeid
user
Fareso
6615f99924c94fd8a22311d2
Atsushi Saito
claimed_verified
2024-04-17T13:15:26.703Z
606b1d984b47ca0dcae61aed
/avatars/2457293c5469f2bc5116f6e915f045c8.svg
AtsushiSaito
user
atsushi3110
6615f99924c94fd8a22311d3
Guangyu Song
6615f99924c94fd8a22311d4
Haoqin Tu
admin_assigned
2024-07-23T14:10:48.584Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
6615f99924c94fd8a22311d5
Stanisław Woźniak
6615f99924c94fd8a22311d6
Ruichong Zhang
claimed_verified
2024-04-10T10:58:34.768Z
6418629fd13ffa408128d7ae
https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png
Zhang Ruichong
user
ZhangRC
6615f99924c94fd8a22311d7
Bingchen Zhao
claimed_verified
2024-06-19T07:39:47.208Z
62dcd71075e9787ec5aa41ba
/avatars/f37ce036b76180ed0fa004f9c8c09363.svg
Bingchen Zhao
user
tennant
6615f99924c94fd8a22311d8
Qihang Zhao
6615f99924c94fd8a22311d9
Peng Zhou
6615f99924c94fd8a22311da
Jian Zhu
6615f99924c94fd8a22311db
Rui-Jie Zhu
claimed_verified
2024-10-14T19:12:40.226Z
63ff09f24852102d4871c19c
https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
Rui-Jie Zhu
user
ridger
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon
the RWKV (RWKV-4) architecture. Our architectural design advancements include
multi-headed matrix-valued states and a dynamic recurrence mechanism that
improve expressivity while maintaining the inference efficiency characteristics
of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a
fast tokenizer based on greedy matching for enhanced multilinguality. We
trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two
Finch models with 1.6 and 3.1 billion parameters and find that they achieve
competitive performance across a wide variety of benchmarks. We release all our
models on HuggingFace under the Apache 2.0 license. Models at:
https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM
Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code
at: https://github.com/RWKV/RWKV-infctx-trainer
Eagle and Finch, improvements over RWKV-4, use multi-headed matrix-valued states and dynamic recurrence for enhanced expressivity and efficiency, trained on a multilingual corpus.
multi-headed matrix-valued states dynamic recurrence mechanism RNNs greedy matching multilingual corpus