<p>Hinge loss is difficult to work with when the derivative is needed because the derivative will be a piece-wise function. <code>max</code> has one non-differentiable point in its solution, and thus the derivative has the same. This was a very prominent issue with non-separable cases of SVM (and a good reason to use ridge regression).</p>

<p>Here's a slide (Original source from <a href="https://sites.google.com/site/ucsdfall2018cogs118a/calendar-notes" rel="nofollow noreferrer">Zhuowen Tu</a>, apologies for the title typo):
<a href="https://i.sstatic.net/hoaGW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hoaGW.png" alt="SVM Hinge Loss equation and derivative"></a></p>

<p>Where hinge loss is defined as <code>max(0, 1-v)</code> and <code>v</code> is the decision boundary of the SVM classifier. More can be found on <a href="https://en.wikipedia.org/wiki/Hinge_loss" rel="nofollow noreferrer">the Hinge Loss Wikipedia</a>.</p>

<p>As for your equation: you can easily pick out the <code>v</code> of the equation, however without more context of those functions it's hard to say how to derive. Unfortunately I don't have access to the paper and cannot guide you any further...</p>

CC BY-SA 4.0
2018-12-18T01:32:39.647Z
2018-12-18T01:32:39.647Z
Alex L
<p>I disagree with the earlier answer that this is difficult to calculate. If we have the function
<span class="math-container">\begin{align*} 
\sum_{t\in\mathcal{T}}  \max \{0, 1 - d(t) \, y(t, \theta)\}
\end{align*}</span>
the gradient with respect to <span class="math-container">$\theta$</span> is
<span class="math-container">\begin{align*} 
&amp; \sum_{t\in\mathcal{T}}g(t) \\
&amp; g(t) := \begin{cases} 
 0 &amp; \text{ if }1 - d(t) y(t, \theta) &lt; 0 \\ 
 -d(t)\dfrac{\partial y}{\partial \theta} &amp; \text{ otherwise} \\ 
\end{cases}
\end{align*}</span>
Theoretically this is ok, it just means that the gradient is not continuous. However, the objective is still continuous assuming that <span class="math-container">$d$</span> and <span class="math-container">$y$</span> are both continuous.</p>
<p>In practice, it's not a problem either. Any automatic differentiation software (tensorflow, pytorch, jax) will handle something like this automatically and correctly.</p>

CC BY-SA 4.0
2022-10-02T01:40:25.963Z
2022-10-02T01:40:25.963Z
Taw
<p>With reference to the research paper entitled <a href="https://ieeexplore.ieee.org/document/7296633" rel="nofollow noreferrer">Sentiment Embeddings with Applications to Sentiment Analysis</a>, I am trying to implement its sentiment ranking model in Python, for which I am required to optimize the following hinge loss function: </p>

<p><span class="math-container">$$\operatorname{loss}_{\text {sRank}}=\sum_{t}^{T} \max \left(0,1-\delta_{s}(t) f_{0}^{\text {rank}}(t)+\delta_{s}(t) f_{1}^{\text {rank}}(t)\right)$$</span></p>

<p>Unlike the usual mean square error, I cannot find its gradient to perform backpropagation.</p>

<p>How do I calculate the gradient of this loss function?</p>

CC BY-SA 4.0
2018-10-08T03:44:01.003Z
You will either need to contact the authors and hope at least one responds with the equation or you need to calculate it yourself.  The following article might help you: https://twice22.github.io/hingeloss/
Brian O'Donnell
<neural-networks><objective-functions><backpropagation><math><gradient-descent>
How do I calculate the gradient of the hinge loss function?
Raj Shrivastava
# How do I calculate the gradient of the hinge loss function?

(Asked by: username_0 on 2018-10-06)

With reference to the research paper entitled [Sentiment Embeddings with Applications to Sentiment Analysis](https://ieeexplore.ieee.org/document/7296633), I am trying to implement its sentiment ranking model in Python, for which I am required to optimize the following hinge loss function:

$$\operatorname{loss}\_{\text {sRank}}=\sum\_{t}^{T} \max \left(0,1-\delta\_{s}(t) f\_{0}^{\text {rank}}(t)+\delta\_{s}(t) f\_{1}^{\text {rank}}(t)\right)$$

Unlike the usual mean square error, I cannot find its gradient to perform backpropagation.

How do I calculate the gradient of this loss function?

**Tags:** neural-networks, objective-functions, backpropagation, math, gradient-descent

> **Comments:**
> **username_1**: You will either need to contact the authors and hope at least one responds with the equation or you need to calculate it yourself.  The following article might help you: https://twice22.github.io/hingeloss/  

---

## 2 answers

### Answer by username_2 on 2018-12-18. Score: 1

Hinge loss is difficult to work with when the derivative is needed because the derivative will be a piece-wise function. `max` has one non-differentiable point in its solution, and thus the derivative has the same. This was a very prominent issue with non-separable cases of SVM (and a good reason to use ridge regression).

Here's a slide (Original source from [Zhuowen Tu](https://sites.google.com/site/ucsdfall2018cogs118a/calendar-notes), apologies for the title typo):
[![SVM Hinge Loss equation and derivative](https://i.sstatic.net/hoaGW.png)](https://i.sstatic.net/hoaGW.png)

Where hinge loss is defined as `max(0, 1-v)` and `v` is the decision boundary of the SVM classifier. More can be found on [the Hinge Loss Wikipedia](https://en.wikipedia.org/wiki/Hinge_loss).

As for your equation: you can easily pick out the `v` of the equation, however without more context of those functions it's hard to say how to derive. Unfortunately I don't have access to the paper and cannot guide you any further...


---

### Answer by username_3 on 2022-10-02. Score: 1

I disagree with the earlier answer that this is difficult to calculate. If we have the function
\begin{align\*}
\sum\_{t\in\mathcal{T}} \max \{0, 1 - d(t) \, y(t, \theta)\}
\end{align\*}
the gradient with respect to $\theta$ is
\begin{align\*}
& \sum\_{t\in\mathcal{T}}g(t) \\
& g(t) := \begin{cases}
0 & \text{ if }1 - d(t) y(t, \theta) < 0 \\
-d(t)\dfrac{\partial y}{\partial \theta} & \text{ otherwise} \\
\end{cases}
\end{align\*}
Theoretically this is ok, it just means that the gradient is not continuous. However, the objective is still continuous assuming that $d$ and $y$ are both continuous.

In practice, it's not a problem either. Any automatic differentiation software (tensorflow, pytorch, jax) will handle something like this automatically and correctly.


---

