title: Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
summary: We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon
  the RWKV (RWKV-4) architecture. Our architectural design advancements include
  multi-headed matrix-valued states and a dynamic recurrence mechanism that
  improve expressivity while maintaining the inference efficiency characteristics
  of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a
  fast tokenizer based on greedy matching for enhanced multilinguality. We
  trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two
  Finch models with 1.6 and 3.1 billion parameters and find that they achieve
  competitive performance across a wide variety of benchmarks. We release all our
  models on HuggingFace under the Apache 2.0 license. Models at:
  https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM
  Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code
  at: https://github.com/RWKV/RWKV-infctx-trainer
paper_authors._id: 6615f99924c94fd8a22311c1
paper_authors.name: Bo Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:49:08.497Z
paper_authors._id: 62b3d8d651b07307bd12b7f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655953609090-noauth.jpeg
paper_authors.fullname: BlinkDL
paper_authors.type: user
paper_authors.user: BlinkDL
paper_authors._id: 6615f99924c94fd8a22311c2
paper_authors.name: Daniel Goldstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:49:10.506Z
paper_authors._id: 647f4bac45baf21ad709fcd0
paper_authors.avatarUrl: /avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
paper_authors.fullname: Dan Goldstein
paper_authors.type: user
paper_authors.user: SmerkyG
paper_authors._id: 6615f99924c94fd8a22311c3
paper_authors.name: Quentin Anthony
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:34:51.007Z
paper_authors._id: 63b9a261efe99543b34e9579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b9a261efe99543b34e9579/aZMhhnurORuPU_WryDUiu.jpeg
paper_authors.fullname: Quentin Anthony
paper_authors.type: user
paper_authors.user: qanthony
paper_authors._id: 6615f99924c94fd8a22311c4
paper_authors.name: Alon Albalak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-15T10:43:23.918Z
paper_authors._id: 611a7ec4289467cafea62d13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
paper_authors.fullname: Alon Albalak
paper_authors.type: user
paper_authors.user: alon-albalak
paper_authors._id: 6615f99924c94fd8a22311c5
paper_authors.name: Eric Alcaide
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-14T12:54:30.685Z
paper_authors._id: 646c8a411ee398a4e940cab1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646c8a411ee398a4e940cab1/5W3dGj8eQqQrxeGPQO1n2.png
paper_authors.fullname: hypnopump
paper_authors.type: user
paper_authors.user: hypnopump
paper_authors._id: 6615f99924c94fd8a22311c6
paper_authors.name: Stella Biderman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:51:30.096Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_authors._id: 6615f99924c94fd8a22311c7
paper_authors.name: Eugene Cheah
paper_authors._id: 6615f99924c94fd8a22311c8
paper_authors.name: Teddy Ferdinan
paper_authors._id: 6615f99924c94fd8a22311c9
paper_authors.name: Haowen Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T19:17:49.238Z
paper_authors._id: 64414d0a66a62c605c94d14d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg
paper_authors.fullname: howard-hou
paper_authors.type: user
paper_authors.user: howard-hou
paper_authors._id: 6615f99924c94fd8a22311ca
paper_authors.name: Przemysław Kazienko
paper_authors._id: 6615f99924c94fd8a22311cb
paper_authors.name: Kranthi Kiran GV
paper_authors._id: 6615f99924c94fd8a22311cc
paper_authors.name: Jan Kocoń
paper_authors._id: 6615f99924c94fd8a22311cd
paper_authors.name: Bartłomiej Koptyra
paper_authors._id: 6615f99924c94fd8a22311ce
paper_authors.name: Satyapriya Krishna
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-23T07:15:39.514Z
paper_authors._id: 6186fef1b1085ab638324e7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6186fef1b1085ab638324e7f/BL6_WJCkxB-BatBUBilT8.jpeg
paper_authors.fullname: Satya
paper_authors.type: user
paper_authors.user: skrishna
paper_authors._id: 6615f99924c94fd8a22311cf
paper_authors.name: Ronald McClelland Jr.
paper_authors._id: 6615f99924c94fd8a22311d0
paper_authors.name: Niklas Muennighoff
paper_authors._id: 6615f99924c94fd8a22311d1
paper_authors.name: Fares Obeid
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:58:45.716Z
paper_authors._id: 643c7c0586ab6dbe34f1eae5
paper_authors.avatarUrl: /avatars/aba29ed4a6092658900cf16f32c90f02.svg
paper_authors.fullname: Fares Obeid
paper_authors.type: user
paper_authors.user: Fareso
paper_authors._id: 6615f99924c94fd8a22311d2
paper_authors.name: Atsushi Saito
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-17T13:15:26.703Z
paper_authors._id: 606b1d984b47ca0dcae61aed
paper_authors.avatarUrl: /avatars/2457293c5469f2bc5116f6e915f045c8.svg
paper_authors.fullname: AtsushiSaito
paper_authors.type: user
paper_authors.user: atsushi3110
paper_authors._id: 6615f99924c94fd8a22311d3
paper_authors.name: Guangyu Song
paper_authors._id: 6615f99924c94fd8a22311d4
paper_authors.name: Haoqin Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T14:10:48.584Z
paper_authors._id: 604ae011caabafacfa48e3de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
paper_authors.fullname: Haoqin Tu
paper_authors.type: user
paper_authors.user: PahaII
paper_authors._id: 6615f99924c94fd8a22311d5
paper_authors.name: Stanisław Woźniak
paper_authors._id: 6615f99924c94fd8a22311d6
paper_authors.name: Ruichong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:58:34.768Z
paper_authors._id: 6418629fd13ffa408128d7ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png
paper_authors.fullname: Zhang Ruichong
paper_authors.type: user
paper_authors.user: ZhangRC
paper_authors._id: 6615f99924c94fd8a22311d7
paper_authors.name: Bingchen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:39:47.208Z
paper_authors._id: 62dcd71075e9787ec5aa41ba
paper_authors.avatarUrl: /avatars/f37ce036b76180ed0fa004f9c8c09363.svg
paper_authors.fullname: Bingchen Zhao
paper_authors.type: user
paper_authors.user: tennant
paper_authors._id: 6615f99924c94fd8a22311d8
paper_authors.name: Qihang Zhao
paper_authors._id: 6615f99924c94fd8a22311d9
paper_authors.name: Peng Zhou
paper_authors._id: 6615f99924c94fd8a22311da
paper_authors.name: Jian Zhu
paper_authors._id: 6615f99924c94fd8a22311db
paper_authors.name: Rui-Jie Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:12:40.226Z
paper_authors._id: 63ff09f24852102d4871c19c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
paper_authors.fullname: Rui-Jie Zhu
paper_authors.type: user
paper_authors.user: ridger
paper_title: Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
paper_summary: We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon
  the RWKV (RWKV-4) architecture. Our architectural design advancements include
  multi-headed matrix-valued states and a dynamic recurrence mechanism that
  improve expressivity while maintaining the inference efficiency characteristics
  of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a
  fast tokenizer based on greedy matching for enhanced multilinguality. We
  trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two
  Finch models with 1.6 and 3.1 billion parameters and find that they achieve
  competitive performance across a wide variety of benchmarks. We release all our
  models on HuggingFace under the Apache 2.0 license. Models at:
  https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM
  Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code
  at: https://github.com/RWKV/RWKV-infctx-trainer
paper_ai_summary: Eagle and Finch, improvements over RWKV-4, use multi-headed matrix-valued states and dynamic recurrence for enhanced expressivity and efficiency, trained on a multilingual corpus.
paper_ai_keywords: multi-headed matrix-valued states dynamic recurrence mechanism RNNs greedy matching multilingual corpus