Answers.Body: <p>Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that trainable parameters receive gradients that point in a direction that decreases the loss over time.</p>
  
  <p>In the absence of a differentiable loss function, the true gradient must be approximated through other methods. For example, in classification problems, the 0-1 loss function is considered the "true" loss, but it is non-convex and difficult to optimize. Instead, <em>surrogate loss functions</em> act as tractable proxies for true loss functions. They are not necessarily worse; negative log-likelihood loss gives a softmax distribution over <span class="math-container">$k$</span> classes rather than just the classification boundary.</p>
  
  <p>For your problem specifically, <span class="math-container">$f(x,a)=min(x,a)$</span> is not a differentiable loss function. It is not differentiable at <span class="math-container">$x=0.5$</span>, but the gradient could be estimated through the <a href="https://en.wikipedia.org/wiki/Subderivative" rel="nofollow noreferrer">subgradient</a>. In practice, this works because neural networks often don't achieve the local/global minima of a loss function but instead asymptotically decreasing values that achieve good generalization error. Tensorflow and PyTorch use subgradients when fed non-differentiable loss functions. You could also use a smooth approximation of the <span class="math-container">$min$</span> function (see <a href="https://mathoverflow.net/questions/35191/a-differentiable-approximation-to-the-minimum-function">this thread</a>) to get better gradients.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T17:37:36.110Z
Answers.LastActivityDate: 2019-01-10T17:37:36.110Z
Answers.UserDisplayName: Shrey
Body: <p>I have non-smooth loss function <span class="math-container">$f(x) = \min(x, 0.5)$</span>.</p>
  <p>Can gradient descent be used for training neural networks with such functions? Can gradient descent be used for fairly general, mathematically not-nice functions?</p>
  <p>PyTorch or TensorFlow can calculate numerically gradients from almost any function, but it is acceptable practice to use general, not-nice loss functions?</p>
  
Tags: <neural-networks><training><objective-functions><gradient-descent>
Title: Can gradient descent training be used for non-smooth loss functions?
UserDisplayName: TomR
ThreadText: # Can gradient descent training be used for non-smooth loss functions?
  
  (Asked by: username_0 on 2019-01-10)
  
  I have non-smooth loss function $f(x) = \min(x, 0.5)$.
  
  Can gradient descent be used for training neural networks with such functions? Can gradient descent be used for fairly general, mathematically not-nice functions?
  
  PyTorch or TensorFlow can calculate numerically gradients from almost any function, but it is acceptable practice to use general, not-nice loss functions?
  
  **Tags:** neural-networks, training, objective-functions, gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-10. Score: 4 | Accepted answer
  
  Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that trainable parameters receive gradients that point in a direction that decreases the loss over time.
  
  In the absence of a differentiable loss function, the true gradient must be approximated through other methods. For example, in classification problems, the 0-1 loss function is considered the "true" loss, but it is non-convex and difficult to optimize. Instead, *surrogate loss functions* act as tractable proxies for true loss functions. They are not necessarily worse; negative log-likelihood loss gives a softmax distribution over $k$ classes rather than just the classification boundary.
  
  For your problem specifically, $f(x,a)=min(x,a)$ is not a differentiable loss function. It is not differentiable at $x=0.5$, but the gradient could be estimated through the [subgradient](https://en.wikipedia.org/wiki/Subderivative). In practice, this works because neural networks often don't achieve the local/global minima of a loss function but instead asymptotically decreasing values that achieve good generalization error. Tensorflow and PyTorch use subgradients when fed non-differentiable loss functions. You could also use a smooth approximation of the $min$ function (see [this thread](https://mathoverflow.net/questions/35191/a-differentiable-approximation-to-the-minimum-function)) to get better gradients.
  
  
  ---
  
  