The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:
text
The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:
new_line
new_line
We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
text
We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
new_line
new_line
🎯 Try the demo and read the write-up here!
text
🎯 Try the demo and read the write-up here!
new_line
https://ruurdkuiper.github.io/tini-lad/
https://ruurdkuiper.github.io/tini-lad/
link
new_line
new_line
Unlike autoregressive (word-for-word) models like ChatGPT, diffusion models iteratively refine a noised sequence. However, most current diffusion approaches rely on all-parameter retraining and repeatedly remasking tokens, which is costly and slow during both training and inference!
text
Unlike autoregressive (word-for-word) models like ChatGPT, diffusion models iteratively refine a noised sequence. However, most current diffusion approaches rely on all-parameter retraining and repeatedly remasking tokens, which is costly and slow during both training and inference!
new_line
new_line
🧠 With LAD:
text
🧠 With LAD:
new_line
- We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
text
- We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
new_line
- Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
text
- Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
new_line
- Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
text
- Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
new_line
new_line
🔍 LAD is built using:
text
🔍 LAD is built using:
new_line
 – A frozen LLaMA-8B backbone
text
 – A frozen LLaMA-8B backbone
new_line
 – Structured noising: token swaps, duplications, replacements, span shifts
text
 – Structured noising: token swaps, duplications, replacements, span shifts
new_line
 – Modified attention masks for bidirectional decoding
text
 – Modified attention masks for bidirectional decoding
new_line
new_line
💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
text
💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
new_line
new_line
The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:

We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.

🎯 Try the demo and read the write-up here!
https://ruurdkuiper.github.io/tini-lad/

Unlike autoregressive (word-for-word) models like ChatGPT, diffusion models iteratively refine a noised sequence. However, most current diffusion approaches rely on all-parameter retraining and repeatedly remasking tokens, which is costly and slow during both training and inference!

🧠 With LAD:
- We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
- Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
- Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!

🔍 LAD is built using:
 – A frozen LLaMA-8B backbone
 – Structured noising: token swaps, duplications, replacements, span shifts
 – Modified attention masks for bidirectional decoding

💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.


🔥
YaTharThShaRma999 John6666 grandhigh catonischnina ddh0 JRizzled Makar7 Ruurd BrandonH
👀
catonischnina qwertyuiopasdfg Ruurd
❤️
ddh0 Ruurd fatemenazery39
👍
UMCU
658a4c914bb41498f7d5e3ca
https://cdn-avatars.huggingface.co/v1/production/uploads/658a4c914bb41498f7d5e3ca/zMJjxfazi9ePc7GZ1jRAE.jpeg
Pro Creations
ProCreations
user
6582dd0229e72d1ea4d10b05
/avatars/89c74635271ec3273d520a59a5857c47.svg
Ruurd Kuiper
Ruurd
user