title: Benchmarking Neural Network Training Algorithms
summary: Training algorithms, broadly construed, are an essential part of every deep
  learning pipeline. Training algorithm improvements that speed up training
  across a wide variety of workloads (e.g., better update rules, tuning
  protocols, learning rate schedules, or data selection schemes) could save time,
  save computational resources, and lead to better, more accurate, models.
  Unfortunately, as a community, we are currently unable to reliably identify
  training algorithm improvements, or even determine the state-of-the-art
  training algorithm. In this work, using concrete experiments, we argue that
  real progress in speeding up training requires new benchmarks that resolve
  three basic challenges faced by empirical comparisons of training algorithms:
  (1) how to decide when training is complete and precisely measure training
  time, (2) how to handle the sensitivity of measurements to exact workload
  details, and (3) how to fairly compare algorithms that require hyperparameter
  tuning. In order to address these challenges, we introduce a new, competitive,
  time-to-result benchmark using multiple workloads running on fixed hardware,
  the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of
  workload variants that make it possible to detect benchmark submissions that
  are more robust to workload changes than current widely-used methods. Finally,
  we evaluate baseline submissions constructed using various optimizers that
  represent current practice, as well as other optimizers that have recently
  received attention in the literature. These baseline results collectively
  demonstrate the feasibility of our benchmark, show that non-trivial gaps
  between methods exist, and set a provisional state-of-the-art for future
  benchmark submissions to try and surpass.
paper_authors._id: 6487d9cc536e4b6067966ea6
paper_authors.name: George E. Dahl
paper_authors._id: 6487d9cc536e4b6067966ea7
paper_authors.name: Frank Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T14:54:19.547Z
paper_authors._id: 6489cfa12ac909eb2b45b990
paper_authors.avatarUrl: /avatars/e04cf5af5dde63f00d24a39f7e3c54fb.svg
paper_authors.fullname: Frank Schneider
paper_authors.type: user
paper_authors.user: FSSchneider
paper_authors._id: 6487d9cc536e4b6067966ea8
paper_authors.name: Zachary Nado
paper_authors._id: 6487d9cc536e4b6067966ea9
paper_authors.name: Naman Agarwal
paper_authors._id: 6487d9cc536e4b6067966eaa
paper_authors.name: Chandramouli Shama Sastry
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-01T01:59:51.973Z
paper_authors._id: 683babb54537373a93801673
paper_authors.avatarUrl: /avatars/d738ec62d3ca8f1d9be3bc56648d6fd4.svg
paper_authors.fullname: Chandramouli Sastry
paper_authors.type: user
paper_authors.user: cssastry
paper_authors._id: 6487d9cc536e4b6067966eab
paper_authors.name: Philipp Hennig
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:39:21.137Z
paper_authors._id: 64885810d3d648eb1f840928
paper_authors.avatarUrl: /avatars/a82c746737179168035db3161e2a7bcd.svg
paper_authors.fullname: Philipp Hennig
paper_authors.type: user
paper_authors.user: PhilippHennig
paper_authors._id: 6487d9cc536e4b6067966eac
paper_authors.name: Sourabh Medapati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:39:11.803Z
paper_authors._id: 648888b165e25cc87b6ab206
paper_authors.avatarUrl: /avatars/c972e4ff17a446c4d2f803fe9315ce12.svg
paper_authors.fullname: Sourabh Medapati
paper_authors.type: user
paper_authors.user: smedapati
paper_authors._id: 6487d9cc536e4b6067966ead
paper_authors.name: Runa Eschenhagen
paper_authors._id: 6487d9cc536e4b6067966eae
paper_authors.name: Priya Kasimbeg
paper_authors._id: 6487d9cc536e4b6067966eaf
paper_authors.name: Daniel Suo
paper_authors._id: 6487d9cc536e4b6067966eb0
paper_authors.name: Juhan Bae
paper_authors._id: 6487d9cc536e4b6067966eb1
paper_authors.name: Justin Gilmer
paper_authors._id: 6487d9cc536e4b6067966eb2
paper_authors.name: Abel L. Peirson
paper_authors._id: 6487d9cc536e4b6067966eb3
paper_authors.name: Bilal Khan
paper_authors._id: 6487d9cc536e4b6067966eb4
paper_authors.name: Rohan Anil
paper_authors._id: 6487d9cc536e4b6067966eb5
paper_authors.name: Mike Rabbat
paper_authors._id: 6487d9cc536e4b6067966eb6
paper_authors.name: Shankar Krishnan
paper_authors._id: 6487d9cc536e4b6067966eb7
paper_authors.name: Daniel Snider
paper_authors._id: 6487d9cc536e4b6067966eb8
paper_authors.name: Ehsan Amid
paper_authors._id: 6487d9cc536e4b6067966eb9
paper_authors.name: Kongtao Chen
paper_authors._id: 6487d9cc536e4b6067966eba
paper_authors.name: Chris J. Maddison
paper_authors._id: 6487d9cc536e4b6067966ebb
paper_authors.name: Rakshith Vasudev
paper_authors._id: 6487d9cc536e4b6067966ebc
paper_authors.name: Michal Badura
paper_authors._id: 6487d9cc536e4b6067966ebd
paper_authors.name: Ankush Garg
paper_authors._id: 6487d9cc536e4b6067966ebe
paper_authors.name: Peter Mattson
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-02T15:02:04.359Z
paper_authors._id: 6570a42253269901a1554106
paper_authors.avatarUrl: /avatars/b33168b6500e152563f60c4d3588cd6d.svg
paper_authors.fullname: Peter Mattson
paper_authors.type: user
paper_authors.user: prmattson
paper_title: Benchmarking Neural Network Training Algorithms
paper_summary: Training algorithms, broadly construed, are an essential part of every deep
  learning pipeline. Training algorithm improvements that speed up training
  across a wide variety of workloads (e.g., better update rules, tuning
  protocols, learning rate schedules, or data selection schemes) could save time,
  save computational resources, and lead to better, more accurate, models.
  Unfortunately, as a community, we are currently unable to reliably identify
  training algorithm improvements, or even determine the state-of-the-art
  training algorithm. In this work, using concrete experiments, we argue that
  real progress in speeding up training requires new benchmarks that resolve
  three basic challenges faced by empirical comparisons of training algorithms:
  (1) how to decide when training is complete and precisely measure training
  time, (2) how to handle the sensitivity of measurements to exact workload
  details, and (3) how to fairly compare algorithms that require hyperparameter
  tuning. In order to address these challenges, we introduce a new, competitive,
  time-to-result benchmark using multiple workloads running on fixed hardware,
  the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of
  workload variants that make it possible to detect benchmark submissions that
  are more robust to workload changes than current widely-used methods. Finally,
  we evaluate baseline submissions constructed using various optimizers that
  represent current practice, as well as other optimizers that have recently
  received attention in the literature. These baseline results collectively
  demonstrate the feasibility of our benchmark, show that non-trivial gaps
  between methods exist, and set a provisional state-of-the-art for future
  benchmark submissions to try and surpass.
paper_ai_summary: A new benchmark, AlgoPerf: Training Algorithms, addresses challenges in evaluating training algorithms by providing a competitive, time-to-result benchmark across workloads and optimizers.
paper_ai_keywords: update rules tuning protocols learning rate schedules data selection schemes training algorithms benchmarks training time hyperparameter tuning optimizers