GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill
  and Extreme KV-Cache Compression
We introduce GoldFinch, a hybrid Linear Attention/Transformer sequence model
that uses a new technique to efficiently generate a highly compressed and
reusable KV-Cache in linear time and space with respect to sequence length.
GoldFinch stacks our new GOLD transformer on top of an enhanced version of the
Finch (RWKV-6) architecture. We train up to 1.5B parameter class models of the
Finch, Llama, and GoldFinch architectures, and find dramatically improved
modeling performance relative to both Finch and Llama. Our cache size savings
increase linearly with model layer count, ranging from 756-2550 times smaller
than the traditional transformer cache for common sizes, enabling inference of
extremely large context lengths even on limited hardware. Although
autoregressive generation has O(n) time complexity per token because of
attention, pre-fill computation of the entire initial cache state for a
submitted context costs only O(1) time per token due to the use of a recurrent
neural network (RNN) to generate this cache. We release our trained weights and
training code under the Apache 2.0 license for community use.
6698c509743d55b416bc9b9d
Daniel Goldstein
claimed_verified
2025-03-19T09:49:03.988Z
647f4bac45baf21ad709fcd0
/avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
Dan Goldstein
user
SmerkyG
6698c509743d55b416bc9b9e
Fares Obeid
claimed_verified
2024-07-18T09:05:06.280Z
643c7c0586ab6dbe34f1eae5
/avatars/aba29ed4a6092658900cf16f32c90f02.svg
Fares Obeid
user
Fareso
6698c509743d55b416bc9b9f
Eric Alcaide
claimed_verified
2024-07-22T07:10:39.751Z
646c8a411ee398a4e940cab1
https://cdn-avatars.huggingface.co/v1/production/uploads/646c8a411ee398a4e940cab1/5W3dGj8eQqQrxeGPQO1n2.png
hypnopump
user
hypnopump
6698c509743d55b416bc9ba0
Guangyu Song
admin_assigned
2024-07-23T14:01:23.143Z
622c062645261ac5cc0bda94
/avatars/ce544b74110f7fe1ad11a3939526f5da.svg
Guangyu Song
user
Guangyu
6698c509743d55b416bc9ba1
Eugene Cheah
admin_assigned
2024-07-23T14:01:09.408Z
6312f1ad3ef0644e2563904b
https://cdn-avatars.huggingface.co/v1/production/uploads/6312f1ad3ef0644e2563904b/DVCsUEeUfynWfdRIqX1Zg.jpeg
Eugene Cheah
user
picocreator
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill
  and Extreme KV-Cache Compression
We introduce GoldFinch, a hybrid Linear Attention/Transformer sequence model
that uses a new technique to efficiently generate a highly compressed and
reusable KV-Cache in linear time and space with respect to sequence length.
GoldFinch stacks our new GOLD transformer on top of an enhanced version of the
Finch (RWKV-6) architecture. We train up to 1.5B parameter class models of the
Finch, Llama, and GoldFinch architectures, and find dramatically improved
modeling performance relative to both Finch and Llama. Our cache size savings
increase linearly with model layer count, ranging from 756-2550 times smaller
than the traditional transformer cache for common sizes, enabling inference of
extremely large context lengths even on limited hardware. Although
autoregressive generation has O(n) time complexity per token because of
attention, pre-fill computation of the entire initial cache state for a
submitted context costs only O(1) time per token due to the use of a recurrent
neural network (RNN) to generate this cache. We release our trained weights and
training code under the Apache 2.0 license for community use.
A hybrid Linear Attention/Transformer model, GoldFinch, efficiently generates a compressed KV-Cache for large context lengths with linear time and space complexity, outperforming previous models.
Linear Attention Transformer KV-Cache GOLD transformer recurrent neural network pre-fill computation O(n) time complexity O(1) time complexity cache size savings extremely large context lengths