title: Retentive Network: A Successor to Transformer for Large Language Models
summary: In this work, we propose Retentive Network (RetNet) as a foundation
  architecture for large language models, simultaneously achieving training
  parallelism, low-cost inference, and good performance. We theoretically derive
  the connection between recurrence and attention. Then we propose the retention
  mechanism for sequence modeling, which supports three computation paradigms,
  i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel
  representation allows for training parallelism. The recurrent representation
  enables low-cost O(1) inference, which improves decoding throughput, latency,
  and GPU memory without sacrificing performance. The chunkwise recurrent
  representation facilitates efficient long-sequence modeling with linear
  complexity, where each chunk is encoded parallelly while recurrently
  summarizing the chunks. Experimental results on language modeling show that
  RetNet achieves favorable scaling results, parallel training, low-cost
  deployment, and efficient inference. The intriguing properties make RetNet a
  strong successor to Transformer for large language models. Code will be
  available at https://aka.ms/retnet.
paper_authors._id: 64b5fb597d9d8f9970f9f7e0
paper_authors.name: Yutao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T07:43:44.054Z
paper_authors._id: 6300ef4779c5ddbc6cf83e1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661005591657-noauth.jpeg
paper_authors.fullname: Yutao Sun
paper_authors.type: user
paper_authors.user: sunyt32
paper_authors._id: 64b5fb597d9d8f9970f9f7e1
paper_authors.name: Li Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-16T09:49:49.350Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 64b5fb597d9d8f9970f9f7e2
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T07:44:52.750Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 64b5fb597d9d8f9970f9f7e3
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T07:45:10.079Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 64b5fb597d9d8f9970f9f7e4
paper_authors.name: Yuqing Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T07:45:25.152Z
paper_authors._id: 6369e01864aad59d4d4501ac
paper_authors.avatarUrl: /avatars/bcbd3f9d0d194eeccd061c4fa6a6e283.svg
paper_authors.fullname: Yuqing Xia
paper_authors.type: user
paper_authors.user: yuqxia
paper_authors._id: 64b5fb597d9d8f9970f9f7e5
paper_authors.name: Jilong Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T07:45:42.578Z
paper_authors._id: 632bfc0e4a4991e7114f2a83
paper_authors.avatarUrl: /avatars/19f8154a30d4787c5ebdf9657e37ea12.svg
paper_authors.fullname: Jilong Xue
paper_authors.type: user
paper_authors.user: jlxue
paper_authors._id: 64b5fb597d9d8f9970f9f7e6
paper_authors.name: Jianyong Wang
paper_authors._id: 64b5fb597d9d8f9970f9f7e7
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T07:46:25.505Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Retentive Network: A Successor to Transformer for Large Language Models
paper_summary: In this work, we propose Retentive Network (RetNet) as a foundation
  architecture for large language models, simultaneously achieving training
  parallelism, low-cost inference, and good performance. We theoretically derive
  the connection between recurrence and attention. Then we propose the retention
  mechanism for sequence modeling, which supports three computation paradigms,
  i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel
  representation allows for training parallelism. The recurrent representation
  enables low-cost O(1) inference, which improves decoding throughput, latency,
  and GPU memory without sacrificing performance. The chunkwise recurrent
  representation facilitates efficient long-sequence modeling with linear
  complexity, where each chunk is encoded parallelly while recurrently
  summarizing the chunks. Experimental results on language modeling show that
  RetNet achieves favorable scaling results, parallel training, low-cost
  deployment, and efficient inference. The intriguing properties make RetNet a
  strong successor to Transformer for large language models. Code will be
  available at https://aka.ms/retnet.
paper_ai_summary: Retentive Network (RetNet) offers a foundation for large language models with parallel training, low-cost inference, and efficient long-sequence modeling through a retention mechanism supporting parallel, recurrent, and chunkwise recurrent paradigms.
paper_ai_keywords: Retentive Network RetNet sequence modeling parallel representation recurrent representation chunkwise recurrent representation Transformer language modeling scaling results parallel training low-cost deployment efficient inference