Benchmarking Neural Network Training Algorithms
Training algorithms, broadly construed, are an essential part of every deep
learning pipeline. Training algorithm improvements that speed up training
across a wide variety of workloads (e.g., better update rules, tuning
protocols, learning rate schedules, or data selection schemes) could save time,
save computational resources, and lead to better, more accurate, models.
Unfortunately, as a community, we are currently unable to reliably identify
training algorithm improvements, or even determine the state-of-the-art
training algorithm. In this work, using concrete experiments, we argue that
real progress in speeding up training requires new benchmarks that resolve
three basic challenges faced by empirical comparisons of training algorithms:
(1) how to decide when training is complete and precisely measure training
time, (2) how to handle the sensitivity of measurements to exact workload
details, and (3) how to fairly compare algorithms that require hyperparameter
tuning. In order to address these challenges, we introduce a new, competitive,
time-to-result benchmark using multiple workloads running on fixed hardware,
the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of
workload variants that make it possible to detect benchmark submissions that
are more robust to workload changes than current widely-used methods. Finally,
we evaluate baseline submissions constructed using various optimizers that
represent current practice, as well as other optimizers that have recently
received attention in the literature. These baseline results collectively
demonstrate the feasibility of our benchmark, show that non-trivial gaps
between methods exist, and set a provisional state-of-the-art for future
benchmark submissions to try and surpass.
6487d9cc536e4b6067966ea6
George E. Dahl
6487d9cc536e4b6067966ea7
Frank Schneider
claimed_verified
2023-06-14T14:54:19.547Z
6489cfa12ac909eb2b45b990
/avatars/e04cf5af5dde63f00d24a39f7e3c54fb.svg
Frank Schneider
user
FSSchneider
6487d9cc536e4b6067966ea8
Zachary Nado
6487d9cc536e4b6067966ea9
Naman Agarwal
6487d9cc536e4b6067966eaa
Chandramouli Shama Sastry
extracted_confirmed
2025-06-01T01:59:51.973Z
683babb54537373a93801673
/avatars/d738ec62d3ca8f1d9be3bc56648d6fd4.svg
Chandramouli Sastry
user
cssastry
6487d9cc536e4b6067966eab
Philipp Hennig
claimed_verified
2023-06-13T17:39:21.137Z
64885810d3d648eb1f840928
/avatars/a82c746737179168035db3161e2a7bcd.svg
Philipp Hennig
user
PhilippHennig
6487d9cc536e4b6067966eac
Sourabh Medapati
claimed_verified
2023-06-13T17:39:11.803Z
648888b165e25cc87b6ab206
/avatars/c972e4ff17a446c4d2f803fe9315ce12.svg
Sourabh Medapati
user
smedapati
6487d9cc536e4b6067966ead
Runa Eschenhagen
6487d9cc536e4b6067966eae
Priya Kasimbeg
6487d9cc536e4b6067966eaf
Daniel Suo
6487d9cc536e4b6067966eb0
Juhan Bae
6487d9cc536e4b6067966eb1
Justin Gilmer
6487d9cc536e4b6067966eb2
Abel L. Peirson
6487d9cc536e4b6067966eb3
Bilal Khan
6487d9cc536e4b6067966eb4
Rohan Anil
6487d9cc536e4b6067966eb5
Mike Rabbat
6487d9cc536e4b6067966eb6
Shankar Krishnan
6487d9cc536e4b6067966eb7
Daniel Snider
6487d9cc536e4b6067966eb8
Ehsan Amid
6487d9cc536e4b6067966eb9
Kongtao Chen
6487d9cc536e4b6067966eba
Chris J. Maddison
6487d9cc536e4b6067966ebb
Rakshith Vasudev
6487d9cc536e4b6067966ebc
Michal Badura
6487d9cc536e4b6067966ebd
Ankush Garg
6487d9cc536e4b6067966ebe
Peter Mattson
extracted_pending
2024-02-02T15:02:04.359Z
6570a42253269901a1554106
/avatars/b33168b6500e152563f60c4d3588cd6d.svg
Peter Mattson
user
prmattson
Benchmarking Neural Network Training Algorithms
Training algorithms, broadly construed, are an essential part of every deep
learning pipeline. Training algorithm improvements that speed up training
across a wide variety of workloads (e.g., better update rules, tuning
protocols, learning rate schedules, or data selection schemes) could save time,
save computational resources, and lead to better, more accurate, models.
Unfortunately, as a community, we are currently unable to reliably identify
training algorithm improvements, or even determine the state-of-the-art
training algorithm. In this work, using concrete experiments, we argue that
real progress in speeding up training requires new benchmarks that resolve
three basic challenges faced by empirical comparisons of training algorithms:
(1) how to decide when training is complete and precisely measure training
time, (2) how to handle the sensitivity of measurements to exact workload
details, and (3) how to fairly compare algorithms that require hyperparameter
tuning. In order to address these challenges, we introduce a new, competitive,
time-to-result benchmark using multiple workloads running on fixed hardware,
the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of
workload variants that make it possible to detect benchmark submissions that
are more robust to workload changes than current widely-used methods. Finally,
we evaluate baseline submissions constructed using various optimizers that
represent current practice, as well as other optimizers that have recently
received attention in the literature. These baseline results collectively
demonstrate the feasibility of our benchmark, show that non-trivial gaps
between methods exist, and set a provisional state-of-the-art for future
benchmark submissions to try and surpass.
A new benchmark, AlgoPerf: Training Algorithms, addresses challenges in evaluating training algorithms by providing a competitive, time-to-result benchmark across workloads and optimizers.
update rules tuning protocols learning rate schedules data selection schemes training algorithms benchmarks training time hyperparameter tuning optimizers