COSMOS: Predictable and Cost-Effective Adaptation of LLMs
Large language models (LLMs) achieve remarkable performance across numerous
tasks by using a diverse array of adaptation strategies. However, optimally
selecting a model and adaptation strategy under resource constraints is
challenging and often requires extensive experimentation. We investigate
whether it is possible to accurately predict both performance and cost without
expensive trials. We formalize the strategy selection problem for LLMs and
introduce COSMOS, a unified prediction framework that efficiently estimates
adaptation outcomes at minimal cost. We instantiate and study the capability of
our framework via a pair of powerful predictors: embedding-augmented
lightweight proxy models to predict fine-tuning performance, and low-sample
scaling laws to forecast retrieval-augmented in-context learning. Extensive
evaluation across eight representative benchmarks demonstrates that COSMOS
achieves high prediction accuracy while reducing computational costs by 92.72%
on average, and up to 98.71% in resource-intensive scenarios. Our results show
that efficient prediction of adaptation outcomes is not only feasible but can
substantially reduce the computational overhead of LLM deployment while
maintaining performance standards.
681d01a340485f29676c6da2
Jiayu Wang
admin_assigned
2025-05-14T12:58:35.540Z
651651f5d93a51ceda3021c3
https://cdn-avatars.huggingface.co/v1/production/uploads/651651f5d93a51ceda3021c3/FE2uGpTKBRWMKTDBv1H-g.png
Jiayu (Mila) Wang
user
MilaWang
681d01a340485f29676c6da3
Aws Albarghouthi
admin_assigned
2025-05-14T12:58:41.276Z
63d04cdb8cf6c8e4d208eaa3
/avatars/756f7237672d26fbf98b1b404536069a.svg
Aws Albarghouthi
user
barghouthi
681d01a340485f29676c6da4
Frederic Sala
admin_assigned
2025-05-14T12:58:48.609Z
63044153fc783bfc74461fa9
https://cdn-avatars.huggingface.co/v1/production/uploads/63044153fc783bfc74461fa9/O5Jz6VPvfVHFyaXH_XqoN.jpeg
Frederic Sala
user
fredsala
COSMOS: Predictable and Cost-Effective Adaptation of LLMs
Large language models (LLMs) achieve remarkable performance across numerous
tasks by using a diverse array of adaptation strategies. However, optimally
selecting a model and adaptation strategy under resource constraints is
challenging and often requires extensive experimentation. We investigate
whether it is possible to accurately predict both performance and cost without
expensive trials. We formalize the strategy selection problem for LLMs and
introduce COSMOS, a unified prediction framework that efficiently estimates
adaptation outcomes at minimal cost. We instantiate and study the capability of
our framework via a pair of powerful predictors: embedding-augmented
lightweight proxy models to predict fine-tuning performance, and low-sample
scaling laws to forecast retrieval-augmented in-context learning. Extensive
evaluation across eight representative benchmarks demonstrates that COSMOS
achieves high prediction accuracy while reducing computational costs by 92.72%
on average, and up to 98.71% in resource-intensive scenarios. Our results show
that efficient prediction of adaptation outcomes is not only feasible but can
substantially reduce the computational overhead of LLM deployment while
maintaining performance standards.
COSMOS, a unified prediction framework, efficiently estimates the outcomes of large language model adaptation strategies, significantly reducing computational costs while maintaining performance.
large language models adaptation strategies COSMOS embedding-augmented lightweight proxy models low-sample scaling laws fine-tuning performance retrieval-augmented in-context learning prediction accuracy computational overhead