Machine Learning, Deep Learning, Artificial Intelligence, Regularization, Deep Learning Book.
reducing the number of steps required to fit the model. The evaluation on the validation set can be done both on another GPU in parallel or done after the epoch. A drawback of weight decay was that we had to manually tweak the weight decay coefficient, which, if chosen wrongly, can lead the model