Hi Sahil!



 Sahil_malik:

would be correct or not to call it a “regularization” method.??


I would not call learning-rate decay regularization.
Of course, we’re discussing semantics – the meaning of the term
“regularization.”  But I think it has a fairly standard definition – or at
least connotation – in this context.
We have a loss function as a function of parameter space, and an
optimization algorithm that seeks to move around in parameter space
to locations with lower losses.  (In practice, with neural networks, we
neither find the global minimum, nor even a local minimum.  We just
want to find a “good enough” location in parameter space with a “low
enough” value of the loss.)
Learning-rate decay is just part of the optimization algorithm, just as
is, say, adding momentum to gradient descent or fancier things like
adaptive momentum.  These don’t change where in parameter space
you want to go – they just try to get you there faster.
Regularization, however, changes the loss function so that different
regions of parameter space are favored.  For example, weight decay,
which we often think of as being part of the optimization algorithm, is
essentially equivalent to adding an L2 regularizer to the loss function.
This means we prefer weights that are smaller, even if they would lead
to a (modestly) larger unregularized loss.  This can be used to prevent
weights from becoming very large and producing numerical instability.
It can also lead to weights that generalize better.  But, to emphasize,
it changes which points in parameter space are favored (by the
regularized loss) relative to other points.
Similarly, one can add sparsity penalties (e.g., L1 regularizers) to the
loss function to prefer having more parameters that are close to zero.
Again, in many instances, such sparser parameter sets generalize
better, even though they have larger unregularized losses computed
for the training data.
To summarize:  Regularization modifies the loss, changing which regions
of parameter space are favored, while things like learning-rate decay
modify the optimization algorithm so that it moves you to the favored
regions of parameter space more efficiently.
Best.
K. Frank