Examples in Machine Learning with Non-Differentiable Objective Functions I was reading the following lecture notes on Gradient Descent and came across the following note:

Supposedly, there are some instances in machine learning where the objective function is non-differentiable.  I was trying to think of some instances in Machine Learning where the Objective Functions are non-differentiable. After doing some thinking and reading about this online, I think one of the instances where this is true is in L1 Regularization (i.e. the absolute value of the model parameters introduces discontinuities):

My Question: Can someone please tell me if my understanding of this concept is correct? In L1 Regularization, the absolute value of the model parameters will likely result in the objective function not being differentiable in the classical sense - and as a result, some alternate method of Gradient Descent will be required to optimize such types of objective functions: thus, we use Subgradient Methods.
Is this correct?
Thanks!
References:

*

*https://raghumeka.github.io/CS289ML/gdnotes.pdf

*https://machinelearningcompass.com/machine_learning_math/subgradient_descent/


In L1 Regularization, the absolute value of the model parameters will likely result in the objective function not being differentiable in the classical sense - and as a result, some alternate method of Gradient Descent will be required ...

You statement is not accurate.

*

*for non-differentiable functions, in many cases, we still can have analytical solutions. Gradient descent is not required for all cases.


*non-differentiable is for specific points. Gradient descent needs the function to be differentiable to runb BUT it does not need the function to be differentiable everywhere. This is because for functions not differentiable at certain points, the only thing we are missing is we do not know how to update x at that point. But nothing prevent us to update x on other points where gradient can be calculated.

Examples: suppose we just want to minimize $f(x)=|x|$. We have an analytical solution of it ($x^*=0$). So gradient descent is not required.
If we want to optimize it using gradient descent, we need the gradient to update the $x$ value in each iteration, so we need the function to be differentiable at a given point.
We have gradient for the function, which is $f'(x)=1, \text{ when } x>0$ and $f'(x)=-1 \text{ when } x<0$. The function is differentiable everywhere except at 0. And we are using it to run the gradient descent.

Here is an additional example in 2D. We have a function $f(x_1,x_2)=|x_1-1.23|+1.7*|x_2-3.21|$, looks like this

It is convex and has global minima at $(1.23, 3.21)$.
The function is NOT differentiable everywhere, but gradient decent works just fine. Following plot shows the gradient decent iterations over time.

