title: RWKV: Reinventing RNNs for the Transformer Era
summary: Transformers have revolutionized almost all natural language processing (NLP)
  tasks but suffer from memory and computational complexity that scales
  quadratically with sequence length. In contrast, recurrent neural networks
  (RNNs) exhibit linear scaling in memory and computational requirements but
  struggle to match the same performance as Transformers due to limitations in
  parallelization and scalability. We propose a novel model architecture,
  Receptance Weighted Key Value (RWKV), that combines the efficient
  parallelizable training of Transformers with the efficient inference of RNNs.
  Our approach leverages a linear attention mechanism and allows us to formulate
  the model as either a Transformer or an RNN, which parallelizes computations
  during training and maintains constant computational and memory complexity
  during inference, leading to the first non-transformer architecture to be
  scaled to tens of billions of parameters. Our experiments reveal that RWKV
  performs on par with similarly sized Transformers, suggesting that future work
  can leverage this architecture to create more efficient models. This work
  presents a significant step towards reconciling the trade-offs between
  computational efficiency and model performance in sequence processing tasks.
paper_authors._id: 646c131ddf2609a541c4a1cc
paper_authors.name: Bo Peng
paper_authors._id: 646c131ddf2609a541c4a1cd
paper_authors.name: Eric Alcaide
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T10:23:58.453Z
paper_authors._id: 646c8a411ee398a4e940cab1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646c8a411ee398a4e940cab1/5W3dGj8eQqQrxeGPQO1n2.png
paper_authors.fullname: hypnopump
paper_authors.type: user
paper_authors.user: hypnopump
paper_authors._id: 646c131ddf2609a541c4a1ce
paper_authors.name: Quentin Anthony
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T07:51:26.034Z
paper_authors._id: 63b9a261efe99543b34e9579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b9a261efe99543b34e9579/aZMhhnurORuPU_WryDUiu.jpeg
paper_authors.fullname: Quentin Anthony
paper_authors.type: user
paper_authors.user: qanthony
paper_authors._id: 646c131ddf2609a541c4a1cf
paper_authors.name: Alon Albalak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T07:51:23.306Z
paper_authors._id: 611a7ec4289467cafea62d13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
paper_authors.fullname: Alon Albalak
paper_authors.type: user
paper_authors.user: alon-albalak
paper_authors._id: 646c131ddf2609a541c4a1d0
paper_authors.name: Samuel Arcadinho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T10:08:40.371Z
paper_authors._id: 60eeedbf50b60c406afc1291
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649111275459-60eeedbf50b60c406afc1291.png
paper_authors.fullname: Samuel Arcadinho
paper_authors.type: user
paper_authors.user: SSamDav
paper_authors._id: 646c131ddf2609a541c4a1d1
paper_authors.name: Huanqi Cao
paper_authors._id: 646c131ddf2609a541c4a1d2
paper_authors.name: Xin Cheng
paper_authors._id: 646c131ddf2609a541c4a1d3
paper_authors.name: Michael Chung
paper_authors._id: 646c131ddf2609a541c4a1d4
paper_authors.name: Matteo Grella
paper_authors._id: 646c131ddf2609a541c4a1d5
paper_authors.name: Kranthi Kiran GV
paper_authors._id: 646c131ddf2609a541c4a1d6
paper_authors.name: Xuzheng He
paper_authors._id: 646c131ddf2609a541c4a1d7
paper_authors.name: Haowen Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T19:17:58.467Z
paper_authors._id: 64414d0a66a62c605c94d14d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg
paper_authors.fullname: howard-hou
paper_authors.type: user
paper_authors.user: howard-hou
paper_authors._id: 646c131ddf2609a541c4a1d8
paper_authors.name: Przemyslaw Kazienko
paper_authors._id: 646c131ddf2609a541c4a1d9
paper_authors.name: Jan Kocon
paper_authors._id: 646c131ddf2609a541c4a1da
paper_authors.name: Jiaming Kong
paper_authors._id: 646c131ddf2609a541c4a1db
paper_authors.name: Bartlomiej Koptyra
paper_authors._id: 646c131ddf2609a541c4a1dc
paper_authors.name: Hayden Lau
paper_authors._id: 646c131ddf2609a541c4a1dd
paper_authors.name: Krishna Sri Ipsit Mantri
paper_authors._id: 646c131ddf2609a541c4a1de
paper_authors.name: Ferdinand Mom
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T08:11:07.996Z
paper_authors._id: 64622c4093f702673bf9b953
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/iB42uASF1DoQth23cizSh.png
paper_authors.fullname: Ferdinand Mom
paper_authors.type: user
paper_authors.user: 3outeille
paper_authors._id: 646c131ddf2609a541c4a1df
paper_authors.name: Atsushi Saito
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-17T13:15:20.653Z
paper_authors._id: 606b1d984b47ca0dcae61aed
paper_authors.avatarUrl: /avatars/2457293c5469f2bc5116f6e915f045c8.svg
paper_authors.fullname: AtsushiSaito
paper_authors.type: user
paper_authors.user: atsushi3110
paper_authors._id: 646c131ddf2609a541c4a1e0
paper_authors.name: Xiangru Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T11:17:39.982Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 646c131ddf2609a541c4a1e1
paper_authors.name: Bolun Wang
paper_authors._id: 646c131ddf2609a541c4a1e2
paper_authors.name: Johan S. Wind
paper_authors._id: 646c131ddf2609a541c4a1e3
paper_authors.name: Stansilaw Wozniak
paper_authors._id: 646c131ddf2609a541c4a1e4
paper_authors.name: Ruichong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T06:39:23.050Z
paper_authors._id: 6418629fd13ffa408128d7ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png
paper_authors.fullname: Zhang Ruichong
paper_authors.type: user
paper_authors.user: ZhangRC
paper_authors._id: 646c131ddf2609a541c4a1e5
paper_authors.name: Zhenyuan Zhang
paper_authors._id: 646c131ddf2609a541c4a1e6
paper_authors.name: Qihang Zhao
paper_authors._id: 646c131ddf2609a541c4a1e7
paper_authors.name: Peng Zhou
paper_authors._id: 646c131ddf2609a541c4a1e8
paper_authors.name: Jian Zhu
paper_authors._id: 646c131ddf2609a541c4a1e9
paper_authors.name: Rui-Jie Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:08:33.084Z
paper_authors._id: 63ff09f24852102d4871c19c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
paper_authors.fullname: Rui-Jie Zhu
paper_authors.type: user
paper_authors.user: ridger
paper_title: RWKV: Reinventing RNNs for the Transformer Era
paper_summary: Transformers have revolutionized almost all natural language processing (NLP)
  tasks but suffer from memory and computational complexity that scales
  quadratically with sequence length. In contrast, recurrent neural networks
  (RNNs) exhibit linear scaling in memory and computational requirements but
  struggle to match the same performance as Transformers due to limitations in
  parallelization and scalability. We propose a novel model architecture,
  Receptance Weighted Key Value (RWKV), that combines the efficient
  parallelizable training of Transformers with the efficient inference of RNNs.
  Our approach leverages a linear attention mechanism and allows us to formulate
  the model as either a Transformer or an RNN, which parallelizes computations
  during training and maintains constant computational and memory complexity
  during inference, leading to the first non-transformer architecture to be
  scaled to tens of billions of parameters. Our experiments reveal that RWKV
  performs on par with similarly sized Transformers, suggesting that future work
  can leverage this architecture to create more efficient models. This work
  presents a significant step towards reconciling the trade-offs between
  computational efficiency and model performance in sequence processing tasks.
paper_ai_summary: A new architecture, RWKV, combines the parallelizable training of Transformers with the efficient inference of RNNs, achieving linear scaling and matching performance.
paper_ai_keywords: Transformers recurrent neural networks (RNNs) linear attention mechanism parallelizable training inference RWKV