Gradient Descent: The Ultimate Optimizer
> We propose to instead learn the hyperparameters themselves by gradient descent, and furthermore to learn the
hyper-hyperparameters by gradient descent as well, and so on ad infinitum.Made me laugh. Can't believe this hasn't been done yet. Maybe I should stop looking at ML research upon a pedestal (regardless, OpenAI's PPO and TRPO papers were amazing, not to mention their Learning with Dexterity one)