Slamming: Training a Speech Language Model on One GPU in a Day
We introduce Slam, a recipe for training high-quality Speech Language Models
(SLMs) on a single academic GPU in 24 hours. We do so through empirical
analysis of model initialisation and architecture, synthetic training data,
preference optimisation with synthetic data and tweaking all other components.
We empirically demonstrate that this training recipe also scales well with more
compute getting results on par with leading SLMs in a fraction of the compute
cost. We hope these insights will make SLM training and research more
accessible. In the context of SLM scaling laws, our results far outperform
predicted compute optimal performance, giving an optimistic view to SLM
feasibility. See code, data, models, samples at -
https://pages.cs.huji.ac.il/adiyoss-lab/slamming .
67bd3972f077ddf1f98bacda
Gallil Maimon
claimed_verified
2025-02-25T09:39:36.258Z
66b9bc2dacdbc1d0b39c3b50
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hwR0pVfP_E8XjimXIxDOU.jpeg
Gallil Maimon
user
gallilmaimon
67bd3972f077ddf1f98bacdb
Avishai Elmakies
claimed_verified
2025-02-25T09:39:33.712Z
644662145004f2cb3af08b27
/avatars/5f2af24c7410a5db46374d0b84fb479d.svg
Avishai Elmakies
user
avishai-elmakies
67bd3972f077ddf1f98bacdc
Yossi Adi
admin_assigned
2025-02-25T16:02:07.243Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
Slamming: Training a Speech Language Model on One GPU in a Day
We introduce Slam, a recipe for training high-quality Speech Language Models
(SLMs) on a single academic GPU in 24 hours. We do so through empirical
analysis of model initialisation and architecture, synthetic training data,
preference optimisation with synthetic data and tweaking all other components.
We empirically demonstrate that this training recipe also scales well with more
compute getting results on par with leading SLMs in a fraction of the compute
cost. We hope these insights will make SLM training and research more
accessible. In the context of SLM scaling laws, our results far outperform
predicted compute optimal performance, giving an optimistic view to SLM
feasibility. See code, data, models, samples at -
https://pages.cs.huji.ac.il/adiyoss-lab/slamming .
A method for training high-quality Speech Language Models on a single GPU within 24 hours, demonstrating cost-effectiveness and outperforming predicted computational scaling laws.
Slam Speech Language Models model initialization architecture synthetic training data preference optimization