title: COSMOS: Predictable and Cost-Effective Adaptation of LLMs
summary: Large language models (LLMs) achieve remarkable performance across numerous
  tasks by using a diverse array of adaptation strategies. However, optimally
  selecting a model and adaptation strategy under resource constraints is
  challenging and often requires extensive experimentation. We investigate
  whether it is possible to accurately predict both performance and cost without
  expensive trials. We formalize the strategy selection problem for LLMs and
  introduce COSMOS, a unified prediction framework that efficiently estimates
  adaptation outcomes at minimal cost. We instantiate and study the capability of
  our framework via a pair of powerful predictors: embedding-augmented
  lightweight proxy models to predict fine-tuning performance, and low-sample
  scaling laws to forecast retrieval-augmented in-context learning. Extensive
  evaluation across eight representative benchmarks demonstrates that COSMOS
  achieves high prediction accuracy while reducing computational costs by 92.72%
  on average, and up to 98.71% in resource-intensive scenarios. Our results show
  that efficient prediction of adaptation outcomes is not only feasible but can
  substantially reduce the computational overhead of LLM deployment while
  maintaining performance standards.
paper_authors._id: 681d01a340485f29676c6da2
paper_authors.name: Jiayu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T12:58:35.540Z
paper_authors._id: 651651f5d93a51ceda3021c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651651f5d93a51ceda3021c3/AjkXG6Z4yveKTZ8NXc5A3.jpeg
paper_authors.fullname: Jiayu (Mila) Wang
paper_authors.type: user
paper_authors.user: MilaWang
paper_authors._id: 681d01a340485f29676c6da3
paper_authors.name: Aws Albarghouthi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T12:58:41.276Z
paper_authors._id: 63d04cdb8cf6c8e4d208eaa3
paper_authors.avatarUrl: /avatars/756f7237672d26fbf98b1b404536069a.svg
paper_authors.fullname: Aws Albarghouthi
paper_authors.type: user
paper_authors.user: barghouthi
paper_authors._id: 681d01a340485f29676c6da4
paper_authors.name: Frederic Sala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T12:58:48.609Z
paper_authors._id: 63044153fc783bfc74461fa9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63044153fc783bfc74461fa9/O5Jz6VPvfVHFyaXH_XqoN.jpeg
paper_authors.fullname: Frederic Sala
paper_authors.type: user
paper_authors.user: fredsala
paper_title: COSMOS: Predictable and Cost-Effective Adaptation of LLMs
paper_summary: Large language models (LLMs) achieve remarkable performance across numerous
  tasks by using a diverse array of adaptation strategies. However, optimally
  selecting a model and adaptation strategy under resource constraints is
  challenging and often requires extensive experimentation. We investigate
  whether it is possible to accurately predict both performance and cost without
  expensive trials. We formalize the strategy selection problem for LLMs and
  introduce COSMOS, a unified prediction framework that efficiently estimates
  adaptation outcomes at minimal cost. We instantiate and study the capability of
  our framework via a pair of powerful predictors: embedding-augmented
  lightweight proxy models to predict fine-tuning performance, and low-sample
  scaling laws to forecast retrieval-augmented in-context learning. Extensive
  evaluation across eight representative benchmarks demonstrates that COSMOS
  achieves high prediction accuracy while reducing computational costs by 92.72%
  on average, and up to 98.71% in resource-intensive scenarios. Our results show
  that efficient prediction of adaptation outcomes is not only feasible but can
  substantially reduce the computational overhead of LLM deployment while
  maintaining performance standards.
paper_ai_summary: COSMOS, a unified prediction framework, efficiently estimates the outcomes of large language model adaptation strategies, significantly reducing computational costs while maintaining performance.
paper_ai_keywords: large language models adaptation strategies COSMOS embedding-augmented lightweight proxy models low-sample scaling laws fine-tuning performance retrieval-augmented in-context learning prediction accuracy computational overhead