Answers.Body: <p><strong>Summary:</strong> the loss needs to be differentiable, with some caveats.</p>
  <hr />
  <p>I will introduce some notation, which I hope is clear: if not I am happy to clarify.</p>
  <p>Consider a neural network with parameters <span class="math-container">$\theta \in \mathbb{R}^d$</span>, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters <span class="math-container">$\theta_\mathrm{min}$</span> which minimise the loss
  function
  <span class="math-container">$$\mathcal{L} \colon \mathbb{R}^d \to \mathbb{R}.$$</span></p>
  <hr />
  <p>If this seems abstract, suppose <span class="math-container">$f(x; \theta)$</span> is the neural network and <span class="math-container">$S = \{(x_i, y_i)\}_{i = 1}^n$</span> is the training set. In binary classification we could have the loss function</p>
  <p><span class="math-container">$$\mathcal{L}(\theta) = \sum_{i = 1}^n \mathbb{1} \{f(x_i; \theta) \ne y_i\} $$</span>
  where <span class="math-container">$\mathbb{1}$</span> is the indicator function which is <span class="math-container">$1$</span> if the condition is satisfied and zero otherwise. I consider the loss function to be a function of
  the <em>parameters</em> and not the data, which is fixed.</p>
  <hr />
  <p>Gradient descent is performed by the update rule
  <span class="math-container">$$ \theta_n \leftarrow \theta_{n - 1} - \gamma \nabla \mathcal{L}(\theta_{n - 1}),$$</span>
  yielding new parameters <span class="math-container">$\theta_n$</span> which should give a smaller loss <span class="math-container">$\mathcal{L}(\theta_n)$</span>. The quantity <span class="math-container">$\gamma$</span> is the familiar <em>learning rate</em>.</p>
  <p>The gradient descent rule requires the <em>gradient</em> <span class="math-container">$\nabla \mathcal{L}(\theta_{n - 1})$</span> to be defined, so the loss function must be differentiable. In most texts on calculus or mathematical analysis you'll find the result that if a function is
  differentiable at a point <span class="math-container">$x$</span>, it is also continuous at <span class="math-container">$x$</span>. Obviously there is no
  hope that we could perform this procedure without knowing the gradient!</p>
  <p>In principle, differentiability is sufficient to run gradient descent. That said, unless <span class="math-container">$\mathcal{L}$</span> is convex, gradient descent offers no guarantees of convergence to a global minimiser. In practice, neural network loss functions are rarely convex
  anyway.</p>
  <p>I have omitted discussion on stochastic gradient descent, but it does not change the requirements for the loss function. There are alternative techniques such as the <a href="https://en.wikipedia.org/wiki/Proximal_gradient_method" rel="nofollow noreferrer">proximal gradient method</a> for non-differentiable functions.</p>
  <p>An unfortunate technicality I have to mention is
  that, strictly speaking, if you use the <span class="math-container">$\mathrm{ReLU}$</span> activation function, the
  neural network function <span class="math-container">$f$</span> becomes non-differentiable. I discuss this further in <a href="https://ai.stackexchange.com/a/28588/44413">this answer</a>. In practice we can assign
  a value and &quot;pretend&quot; <span class="math-container">$\mathrm{ReLU}$</span> is differentiable everywhere.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-28T10:46:51.140Z
Answers.LastActivityDate: 2021-07-28T10:46:51.140Z
Answers.UserDisplayName: htl
Body: <p>Loss functions are used in training neural networks.</p>
  <p>I am interested in knowing the mathematical properties that are necessary for a loss function to <strong>participate in gradient descent optimization</strong>.</p>
  <p>I know some <em>possible</em> candidates that may decide whether a function can be a loss function or not. They include</p>
  <ol>
  <li>Continuous at every point in <span class="math-container">$\mathbb{R}$</span></li>
  <li>Differentiable at every point in <span class="math-container">$\mathbb{R}$</span></li>
  </ol>
  <p>But, I am not sure whether these two properties are necessary for a function to become a loss function.</p>
  <p>Are these two properties <strong>necessary</strong>?  Are there any <strong>other mathematical properties that are necessary</strong> for a function to become a loss function to participate in gradient descent optimization?</p>
  <p>Note that this question is not asking for recommended properties for a loss function. Asking only the mandatory properties in a given context.</p>
  
Tags: <deep-learning><objective-functions><math><optimization><gradient-descent>
Title: What are the necessary mathematical properties to be a loss function in gradient based optimizations?
UserDisplayName: hanugm
ThreadText: # What are the necessary mathematical properties to be a loss function in gradient based optimizations?
  
  (Asked by: username_0 on 2021-07-28)
  
  Loss functions are used in training neural networks.
  
  I am interested in knowing the mathematical properties that are necessary for a loss function to **participate in gradient descent optimization**.
  
  I know some *possible* candidates that may decide whether a function can be a loss function or not. They include
  
  1. Continuous at every point in $\mathbb{R}$
  2. Differentiable at every point in $\mathbb{R}$
  
  But, I am not sure whether these two properties are necessary for a function to become a loss function.
  
  Are these two properties **necessary**? Are there any **other mathematical properties that are necessary** for a function to become a loss function to participate in gradient descent optimization?
  
  Note that this question is not asking for recommended properties for a loss function. Asking only the mandatory properties in a given context.
  
  **Tags:** deep-learning, objective-functions, math, optimization, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-28. Score: 3
  
  **Summary:** the loss needs to be differentiable, with some caveats.
  
  ---
  
  I will introduce some notation, which I hope is clear: if not I am happy to clarify.
  
  Consider a neural network with parameters $\theta \in \mathbb{R}^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta\_\mathrm{min}$ which minimise the loss
  function
  $$\mathcal{L} \colon \mathbb{R}^d \to \mathbb{R}.$$
  
  ---
  
  If this seems abstract, suppose $f(x; \theta)$ is the neural network and $S = \{(x\_i, y\_i)\}\_{i = 1}^n$ is the training set. In binary classification we could have the loss function
  
  $$\mathcal{L}(\theta) = \sum\_{i = 1}^n \mathbb{1} \{f(x\_i; \theta) \ne y\_i\} $$
  where $\mathbb{1}$ is the indicator function which is $1$ if the condition is satisfied and zero otherwise. I consider the loss function to be a function of
  the *parameters* and not the data, which is fixed.
  
  ---
  
  Gradient descent is performed by the update rule
  $$ \theta\_n \leftarrow \theta\_{n - 1} - \gamma \nabla \mathcal{L}(\theta\_{n - 1}),$$
  yielding new parameters $\theta\_n$ which should give a smaller loss $\mathcal{L}(\theta\_n)$. The quantity $\gamma$ is the familiar *learning rate*.
  
  The gradient descent rule requires the *gradient* $\nabla \mathcal{L}(\theta\_{n - 1})$ to be defined, so the loss function must be differentiable. In most texts on calculus or mathematical analysis you'll find the result that if a function is
  differentiable at a point $x$, it is also continuous at $x$. Obviously there is no
  hope that we could perform this procedure without knowing the gradient!
  
  In principle, differentiability is sufficient to run gradient descent. That said, unless $\mathcal{L}$ is convex, gradient descent offers no guarantees of convergence to a global minimiser. In practice, neural network loss functions are rarely convex
  anyway.
  
  I have omitted discussion on stochastic gradient descent, but it does not change the requirements for the loss function. There are alternative techniques such as the [proximal gradient method](https://en.wikipedia.org/wiki/Proximal_gradient_method) for non-differentiable functions.
  
  An unfortunate technicality I have to mention is
  that, strictly speaking, if you use the $\mathrm{ReLU}$ activation function, the
  neural network function $f$ becomes non-differentiable. I discuss this further in [this answer](https://ai.stackexchange.com/a/28588/44413). In practice we can assign
  a value and "pretend" $\mathrm{ReLU}$ is differentiable everywhere.
  
  
  ---
  
  