Artificial Intelligence, Deep Learning, Machine Learning, Gradient Descent, Optimization.
not zero most of the times, for such parameters we want to have a smaller learning rate in contrast if we have a sparse parameter which will be off (zero) most of the times that mean the derivative will be zero most of the times. For such feature whenever this feature is on (not zero) we want to