Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank
adaptation (LoRA) reduce the cost of customizing large language models (LLMs),
yet still require a separate optimization run for every downstream dataset. We
introduce Drag-and-Drop LLMs (\textit{DnD)}, a prompt-conditioned
parameter generator that eliminates per-task training by mapping a handful of
unlabeled task prompts directly to LoRA weight updates. A lightweight text
encoder distills each prompt batch into condition embeddings, which are then
transformed by a cascaded hyper-convolutional decoder into the full set of LoRA
matrices. Once trained in a diverse collection of prompt-checkpoint pairs, DnD
produces task-specific parameters in seconds, yielding i) up to
12,000times lower overhead than full fine-tuning, ii) average gains
up to 30\% in performance over the strongest training LoRAs on unseen
common-sense reasoning, math, coding, and multimodal benchmarks, and iii)
robust cross-domain generalization despite never seeing the target data or
labels. Our results demonstrate that prompt-conditioned parameter generation is
a viable alternative to gradient-based adaptation for rapidly specializing
LLMs. Our project is available at
https://jerryliang24.github.io/DnD{https://jerryliang24.github.io/DnD}.
6858d099c0c8e29df8ea3ccb
Zhiyuan Liang
claimed_verified
2025-06-24T08:09:53.376Z
650842d114302b1d76e429d4
/avatars/27bd94f66c63bb08ef176751a5ac9dd5.svg
Zhiyuan Liang
user
Jerrylz
6858d099c0c8e29df8ea3ccc
Dongwen Tang
6858d099c0c8e29df8ea3ccd
Yuhao Zhou
6858d099c0c8e29df8ea3cce
Xuanlei Zhao
claimed_verified
2025-06-24T08:09:55.890Z
64f5937db6d7050b19c68fec
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5937db6d7050b19c68fec/4lWYw6-VxZsbl-1Rd9FSy.jpeg
Xuanlei Zhao
user
oahzxl
6858d099c0c8e29df8ea3ccf
Mingjia Shi
claimed_verified
2025-06-25T08:09:32.549Z
654b28c27f0fd3f254a50314
/avatars/cf4ae1e8db21c0d7e0381b2784447f23.svg
Mingjia Shi
user
BDeM
6858d099c0c8e29df8ea3cd0
Wangbo Zhao
6858d099c0c8e29df8ea3cd1
Zekai Li
6858d099c0c8e29df8ea3cd2
Peihao Wang
6858d099c0c8e29df8ea3cd3
Konstantin Schürholt
6858d099c0c8e29df8ea3cd4
Damian Borth
6858d099c0c8e29df8ea3cd5
Michael M. Bronstein
6858d099c0c8e29df8ea3cd6
Yang You
6858d099c0c8e29df8ea3cd7
Zhangyang Wang
6858d099c0c8e29df8ea3cd8
Kai Wang
claimed_verified
2025-06-23T08:14:35.657Z
655452b8432af1b1116394d1
/avatars/85860fb3c2d09c9c23e7677d7129cca3.svg
Kai Wang
user
VictorKai1996NUS
Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank
adaptation (LoRA) reduce the cost of customizing large language models (LLMs),
yet still require a separate optimization run for every downstream dataset. We
introduce Drag-and-Drop LLMs (\textit{DnD)}, a prompt-conditioned
parameter generator that eliminates per-task training by mapping a handful of
unlabeled task prompts directly to LoRA weight updates. A lightweight text
encoder distills each prompt batch into condition embeddings, which are then
transformed by a cascaded hyper-convolutional decoder into the full set of LoRA
matrices. Once trained in a diverse collection of prompt-checkpoint pairs, DnD
produces task-specific parameters in seconds, yielding i) up to
12,000times lower overhead than full fine-tuning, ii) average gains
up to 30\% in performance over the strongest training LoRAs on unseen
common-sense reasoning, math, coding, and multimodal benchmarks, and iii)
robust cross-domain generalization despite never seeing the target data or
labels. Our results demonstrate that prompt-conditioned parameter generation is
a viable alternative to gradient-based adaptation for rapidly specializing
LLMs. Our project is available at
https://jerryliang24.github.io/DnD{https://jerryliang24.github.io/DnD}.
Drag-and-Drop LLMs generate task-specific parameters through prompt-conditioned parameter generation, achieving significant efficiency gains and cross-domain generalization without per-task training.
Parameter-Efficient Fine-Tuning PEFT low-rank adaptation LoRA large language models prompts condition embeddings hyper-convolutional decoder LoRA matrices common-sense reasoning math coding multimodal benchmarks