Machine Learning, Deep Learning, Artificial Intelligence, AI, Neural Networks.
Regularization We normally don’t regularize bias and regularize weights only. We can use hessian matrix and it’s eigenvalues and eigenvectors to see the sensitivity of the weights to the weight decay. The weight wi will be rescaled using (λi/λi + α) where λi (eigenvalue) measures the sensitivity of