title: RADLADS: Rapid Attention Distillation to Linear Attention Decoders at
    Scale
summary: We present Rapid Attention Distillation to Linear Attention Decoders at Scale
  (RADLADS), a protocol for rapidly converting softmax attention transformers
  into linear attention decoder models, along with two new RWKV-variant
  architectures, and models converted from popular Qwen2.5 open source models in
  7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens,
  less than 0.005% of the token count used to train the original teacher models.
  Converting to our 72B linear attention model costs less than \$2,000 USD at
  today's prices, yet quality at inference remains close to the original
  transformer. These models achieve state-of-the-art downstream performance
  across a set of standard benchmarks for linear attention models of their size.
  We release all our models on HuggingFace under the Apache 2.0 license, with the
  exception of our 72B models which are also governed by the Qwen License
  Agreement.
    Models at
  https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102
  Training Code at https://github.com/recursal/RADLADS-paper
paper_authors._id: 681ac8a09f4ed2ece10fac18
paper_authors.name: Daniel Goldstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-07T08:08:53.268Z
paper_authors._id: 647f4bac45baf21ad709fcd0
paper_authors.avatarUrl: /avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
paper_authors.fullname: Dan Goldstein
paper_authors.type: user
paper_authors.user: SmerkyG
paper_authors._id: 681ac8a09f4ed2ece10fac19
paper_authors.name: Eric Alcaide
paper_authors._id: 681ac8a09f4ed2ece10fac1a
paper_authors.name: Janna Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T08:59:35.127Z
paper_authors._id: 66b3d98e040c500914ef558f
paper_authors.avatarUrl: /avatars/a90f8306dbd7747520ce5b941ee3bbcb.svg
paper_authors.fullname: Janna
paper_authors.type: user
paper_authors.user: jannalu
paper_authors._id: 681ac8a09f4ed2ece10fac1b
paper_authors.name: Eugene Cheah
paper_title: RADLADS: Rapid Attention Distillation to Linear Attention Decoders at
    Scale
paper_summary: We present Rapid Attention Distillation to Linear Attention Decoders at Scale
  (RADLADS), a protocol for rapidly converting softmax attention transformers
  into linear attention decoder models, along with two new RWKV-variant
  architectures, and models converted from popular Qwen2.5 open source models in
  7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens,
  less than 0.005% of the token count used to train the original teacher models.
  Converting to our 72B linear attention model costs less than \$2,000 USD at
  today's prices, yet quality at inference remains close to the original
  transformer. These models achieve state-of-the-art downstream performance
  across a set of standard benchmarks for linear attention models of their size.
  We release all our models on HuggingFace under the Apache 2.0 license, with the
  exception of our 72B models which are also governed by the Qwen License
  Agreement.
    Models at
  https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102
  Training Code at https://github.com/recursal/RADLADS-paper
paper_ai_summary: A protocol converts softmax attention transformers into linear attention decoders using minimal tokens while maintaining quality and performance.
paper_ai_keywords: softmax attention transformers linear attention decoders Rapid Attention Distillation RADLADS RWKV-variant architectures Qwen2.5 models HuggingFace Apache 2.0 license Qwen License Agreement