Hi Sahil!



 Sahil_malik:

Would you consider then LRReduceOnPlateau and EarlyStopping as regularization


Again, it’s a question of semantics, but I would not call techniques such
as early stopping and the algorithm in pytorch’s ReduceLROnPlateau
kinds of regularization.  They are overlays on top of the optimization
algorithm that take a peek at some validation results.  But they don’t
in any substantive sense affect the regions in parameter towards which
the optimization algorithm moves you.
That is, in my language, the weights in the network are not being in
any way regularized.
Regularization can be used to reduce overfitting, but I would not say
that anything that reduces overfitting is regularization.  For example,
training on more data (or sometimes on augmented data) will reduce
overfitting (and will move you towards different regions of parameter
space), but I would not call training on more data “regularization.”
Best.
K. Frank