title: Prioritized training on points that are learnable, worth learning, and not yet learned
abstract: We introduce reducible held-out loss selection (RHOLS), a technique for faster model training which selects a sequence of training points that are “just right”. We propose a tractable information-theoretic acquisition function—the reducible heldout loss—to efficiently choose training points that maximize information about a holdout set. We show that the “hard” (e.g. high loss) points usually selected in the optimization literature are typically noisy, leading to deterioration on real-world datasets. At the same time, “easy” (e.g. low noise) samples, often prioritized for curriculum learning, confer less information. In contrast, RHOLS chooses points that are “just right” and trains in fewer steps than the above approaches.
authors: Sören Mindermann Muhammed Razzak Mrinank Sharma Jan M. Brauner Winnie Xu Andreas Kirsch Aidan Gomez Benedikt Höltgen Sebastian Farquhar Yarin Gal
hIndex: Yarin Gal 38