title: GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill
    and Extreme KV-Cache Compression
summary: We introduce GoldFinch, a hybrid Linear Attention/Transformer sequence model
  that uses a new technique to efficiently generate a highly compressed and
  reusable KV-Cache in linear time and space with respect to sequence length.
  GoldFinch stacks our new GOLD transformer on top of an enhanced version of the
  Finch (RWKV-6) architecture. We train up to 1.5B parameter class models of the
  Finch, Llama, and GoldFinch architectures, and find dramatically improved
  modeling performance relative to both Finch and Llama. Our cache size savings
  increase linearly with model layer count, ranging from 756-2550 times smaller
  than the traditional transformer cache for common sizes, enabling inference of
  extremely large context lengths even on limited hardware. Although
  autoregressive generation has O(n) time complexity per token because of
  attention, pre-fill computation of the entire initial cache state for a
  submitted context costs only O(1) time per token due to the use of a recurrent
  neural network (RNN) to generate this cache. We release our trained weights and
  training code under the Apache 2.0 license for community use.
paper_authors._id: 6698c509743d55b416bc9b9d
paper_authors.name: Daniel Goldstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:49:03.988Z
paper_authors._id: 647f4bac45baf21ad709fcd0
paper_authors.avatarUrl: /avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
paper_authors.fullname: Dan Goldstein
paper_authors.type: user
paper_authors.user: SmerkyG
paper_authors._id: 6698c509743d55b416bc9b9e
paper_authors.name: Fares Obeid
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:06.280Z
paper_authors._id: 643c7c0586ab6dbe34f1eae5
paper_authors.avatarUrl: /avatars/aba29ed4a6092658900cf16f32c90f02.svg
paper_authors.fullname: Fares Obeid
paper_authors.type: user
paper_authors.user: Fareso
paper_authors._id: 6698c509743d55b416bc9b9f
paper_authors.name: Eric Alcaide
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:39.751Z
paper_authors._id: 646c8a411ee398a4e940cab1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646c8a411ee398a4e940cab1/5W3dGj8eQqQrxeGPQO1n2.png
paper_authors.fullname: hypnopump
paper_authors.type: user
paper_authors.user: hypnopump
paper_authors._id: 6698c509743d55b416bc9ba0
paper_authors.name: Guangyu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T14:01:23.143Z
paper_authors._id: 622c062645261ac5cc0bda94
paper_authors.avatarUrl: /avatars/ce544b74110f7fe1ad11a3939526f5da.svg
paper_authors.fullname: Guangyu Song
paper_authors.type: user
paper_authors.user: Guangyu
paper_authors._id: 6698c509743d55b416bc9ba1
paper_authors.name: Eugene Cheah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T14:01:09.408Z
paper_authors._id: 6312f1ad3ef0644e2563904b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6312f1ad3ef0644e2563904b/DVCsUEeUfynWfdRIqX1Zg.jpeg
paper_authors.fullname: Eugene Cheah
paper_authors.type: user
paper_authors.user: picocreator
paper_title: GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill
    and Extreme KV-Cache Compression
paper_summary: We introduce GoldFinch, a hybrid Linear Attention/Transformer sequence model
  that uses a new technique to efficiently generate a highly compressed and
  reusable KV-Cache in linear time and space with respect to sequence length.
  GoldFinch stacks our new GOLD transformer on top of an enhanced version of the
  Finch (RWKV-6) architecture. We train up to 1.5B parameter class models of the
  Finch, Llama, and GoldFinch architectures, and find dramatically improved
  modeling performance relative to both Finch and Llama. Our cache size savings
  increase linearly with model layer count, ranging from 756-2550 times smaller
  than the traditional transformer cache for common sizes, enabling inference of
  extremely large context lengths even on limited hardware. Although
  autoregressive generation has O(n) time complexity per token because of
  attention, pre-fill computation of the entire initial cache state for a
  submitted context costs only O(1) time per token due to the use of a recurrent
  neural network (RNN) to generate this cache. We release our trained weights and
  training code under the Apache 2.0 license for community use.
paper_ai_summary: A hybrid Linear Attention/Transformer model, GoldFinch, efficiently generates a compressed KV-Cache for large context lengths with linear time and space complexity, outperforming previous models.
paper_ai_keywords: Linear Attention Transformer KV-Cache GOLD transformer recurrent neural network pre-fill computation O(n) time complexity O(1) time complexity cache size savings extremely large context lengths