content.raw: The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:
content.type: text
content.value: The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:
content.type: new_line
content.type: new_line
content.raw: We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
content.type: text
content.value: We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
content.type: new_line
content.type: new_line
content.raw: Unlike autoregressive models, diffusion-style generation iteratively refines a noised sequence. However, most current diffusion approaches rely on all-parameter re-training and repeatedly remasking tokens, which is costly and slow during both training and inference!
content.type: text
content.value: Unlike autoregressive models, diffusion-style generation iteratively refines a noised sequence. However, most current diffusion approaches rely on all-parameter re-training and repeatedly remasking tokens, which is costly and slow during both training and inference!
content.type: new_line
content.type: new_line
content.raw: 🧠 With LAD:
content.type: text
content.value: 🧠 With LAD:
content.type: new_line
content.raw: - We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
content.type: text
content.value: - We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
content.type: new_line
content.raw: - Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
content.type: text
content.value: - Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
content.type: new_line
content.raw: - Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
content.type: text
content.value: - Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
content.type: new_line
content.type: new_line
content.raw: 🔍 LAD is built using:
content.type: text
content.value: 🔍 LAD is built using:
content.type: new_line
content.raw:  – A frozen LLaMA-8B backbone
content.type: text
content.value:  – A frozen LLaMA-8B backbone
content.type: new_line
content.raw:  – Structured noising: token swaps, duplications, replacements, span shifts
content.type: text
content.value:  – Structured noising: token swaps, duplications, replacements, span shifts
content.type: new_line
content.raw:  – Modified attention masks for bidirectional decoding
content.type: text
content.value:  – Modified attention masks for bidirectional decoding
content.type: new_line
content.type: new_line
content.raw: 💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
content.type: text
content.value: 💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
content.type: new_line
content.type: new_line
content.raw: 🎯 Try the demo and read the write-up here:
content.type: text
content.value: 🎯 Try the demo and read the write-up here:
content.type: new_line
content.href: https://ruurdkuiper.github.io/tini-lad/
content.raw: https://ruurdkuiper.github.io/tini-lad/
content.type: link
content.type: new_line
content.type: new_line
rawContent: The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:
  
  We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
  
  Unlike autoregressive models, diffusion-style generation iteratively refines a noised sequence. However, most current diffusion approaches rely on all-parameter re-training and repeatedly remasking tokens, which is costly and slow during both training and inference!
  
  🧠 With LAD:
  - We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
  - Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
  - Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
  
  🔍 LAD is built using:
   – A frozen LLaMA-8B backbone
   – Structured noising: token swaps, duplications, replacements, span shifts
   – Modified attention masks for bidirectional decoding
  
  💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
  
  🎯 Try the demo and read the write-up here:
  https://ruurdkuiper.github.io/tini-lad/
  
  
reactions.reaction: 🔥
reactions.users: YaTharThShaRma999