title: Tune As You Scale: Hyperparameter Optimization For Compute Efficient
    Training
summary: Hyperparameter tuning of deep learning models can lead to order-of-magnitude
  performance gains for the same amount of compute. Despite this, systematic
  tuning is uncommon, particularly for large models, which are expensive to
  evaluate and tend to have many hyperparameters, necessitating difficult
  judgment calls about tradeoffs, budgets, and search bounds. To address these
  issues and propose a practical method for robustly tuning large models, we
  present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian
  optimization algorithm that performs local search around the performance-cost
  Pareto frontier. CARBS does well even in unbounded search spaces with many
  hyperparameters, learns scaling relationships so that it can tune models even
  as they are scaled up, and automates much of the "black magic" of tuning. Among
  our results, we effectively solve the entire ProcGen benchmark just by tuning a
  simple baseline (PPO, as provided in the original ProcGen paper). We also
  reproduce the model size vs. training tokens scaling result from the Chinchilla
  project (Hoffmann et al. 2022), while simultaneously discovering scaling laws
  for every other hyperparameter, via an easy automated process that uses
  significantly less compute and is applicable to any deep learning problem (not
  just language models).
paper_authors._id: 648bc57995d85bfac02180d9
paper_authors.name: Abraham J. Fetterman
paper_authors._id: 648bc57995d85bfac02180da
paper_authors.name: Ellie Kitanidis
paper_authors._id: 648bc57995d85bfac02180db
paper_authors.name: Joshua Albrecht
paper_authors._id: 648bc57995d85bfac02180dc
paper_authors.name: Zachary Polizzi
paper_authors._id: 648bc57995d85bfac02180dd
paper_authors.name: Bryden Fogelman
paper_authors._id: 648bc57995d85bfac02180de
paper_authors.name: Maksis Knutins
paper_authors._id: 648bc57995d85bfac02180df
paper_authors.name: Bartosz Wróblewski
paper_authors._id: 648bc57995d85bfac02180e0
paper_authors.name: James B. Simon
paper_authors._id: 648bc57995d85bfac02180e1
paper_authors.name: Kanjun Qiu
paper_title: Tune As You Scale: Hyperparameter Optimization For Compute Efficient
    Training
paper_summary: Hyperparameter tuning of deep learning models can lead to order-of-magnitude
  performance gains for the same amount of compute. Despite this, systematic
  tuning is uncommon, particularly for large models, which are expensive to
  evaluate and tend to have many hyperparameters, necessitating difficult
  judgment calls about tradeoffs, budgets, and search bounds. To address these
  issues and propose a practical method for robustly tuning large models, we
  present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian
  optimization algorithm that performs local search around the performance-cost
  Pareto frontier. CARBS does well even in unbounded search spaces with many
  hyperparameters, learns scaling relationships so that it can tune models even
  as they are scaled up, and automates much of the "black magic" of tuning. Among
  our results, we effectively solve the entire ProcGen benchmark just by tuning a
  simple baseline (PPO, as provided in the original ProcGen paper). We also
  reproduce the model size vs. training tokens scaling result from the Chinchilla
  project (Hoffmann et al. 2022), while simultaneously discovering scaling laws
  for every other hyperparameter, via an easy automated process that uses
  significantly less compute and is applicable to any deep learning problem (not
  just language models).
paper_ai_keywords: Bayesian optimization Pareto frontier performance-cost Pareto frontier local search unbounded search spaces Bayesian Search Cost-Aware Pareto Region Bayesian Search (CARBS) scaling relationships ProcGen benchmark Proximal Policy Optimization (PPO) model size vs. training tokens scaling Chinchilla project