Answers.Body: <p>Hinge loss is difficult to work with when the derivative is needed because the derivative will be a piece-wise function. <code>max</code> has one non-differentiable point in its solution, and thus the derivative has the same. This was a very prominent issue with non-separable cases of SVM (and a good reason to use ridge regression).</p>
  
  <p>Here's a slide (Original source from <a href="https://sites.google.com/site/ucsdfall2018cogs118a/calendar-notes" rel="nofollow noreferrer">Zhuowen Tu</a>, apologies for the title typo):
  <a href="https://i.sstatic.net/hoaGW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hoaGW.png" alt="SVM Hinge Loss equation and derivative"></a></p>
  
  <p>Where hinge loss is defined as <code>max(0, 1-v)</code> and <code>v</code> is the decision boundary of the SVM classifier. More can be found on <a href="https://en.wikipedia.org/wiki/Hinge_loss" rel="nofollow noreferrer">the Hinge Loss Wikipedia</a>.</p>
  
  <p>As for your equation: you can easily pick out the <code>v</code> of the equation, however without more context of those functions it's hard to say how to derive. Unfortunately I don't have access to the paper and cannot guide you any further...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-18T01:32:39.647Z
Answers.LastActivityDate: 2018-12-18T01:32:39.647Z
Answers.UserDisplayName: Alex L
Answers.Body: <p>I disagree with the earlier answer that this is difficult to calculate. If we have the function
  <span class="math-container">\begin{align*} 
  \sum_{t\in\mathcal{T}}  \max \{0, 1 - d(t) \, y(t, \theta)\}
  \end{align*}</span>
  the gradient with respect to <span class="math-container">$\theta$</span> is
  <span class="math-container">\begin{align*} 
  &amp; \sum_{t\in\mathcal{T}}g(t) \\
  &amp; g(t) := \begin{cases} 
   0 &amp; \text{ if }1 - d(t) y(t, \theta) &lt; 0 \\ 
   -d(t)\dfrac{\partial y}{\partial \theta} &amp; \text{ otherwise} \\ 
  \end{cases}
  \end{align*}</span>
  Theoretically this is ok, it just means that the gradient is not continuous. However, the objective is still continuous assuming that <span class="math-container">$d$</span> and <span class="math-container">$y$</span> are both continuous.</p>
  <p>In practice, it's not a problem either. Any automatic differentiation software (tensorflow, pytorch, jax) will handle something like this automatically and correctly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-02T01:40:25.963Z
Answers.LastActivityDate: 2022-10-02T01:40:25.963Z
Answers.UserDisplayName: Taw
Body: <p>With reference to the research paper entitled <a href="https://ieeexplore.ieee.org/document/7296633" rel="nofollow noreferrer">Sentiment Embeddings with Applications to Sentiment Analysis</a>, I am trying to implement its sentiment ranking model in Python, for which I am required to optimize the following hinge loss function: </p>
  
  <p><span class="math-container">$$\operatorname{loss}_{\text {sRank}}=\sum_{t}^{T} \max \left(0,1-\delta_{s}(t) f_{0}^{\text {rank}}(t)+\delta_{s}(t) f_{1}^{\text {rank}}(t)\right)$$</span></p>
  
  <p>Unlike the usual mean square error, I cannot find its gradient to perform backpropagation.</p>
  
  <p>How do I calculate the gradient of this loss function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-08T03:44:01.003Z
Comments.Text: You will either need to contact the authors and hope at least one responds with the equation or you need to calculate it yourself.  The following article might help you: https://twice22.github.io/hingeloss/
Comments.UserDisplayName: Brian O'Donnell
Tags: <neural-networks><objective-functions><backpropagation><math><gradient-descent>
Title: How do I calculate the gradient of the hinge loss function?
UserDisplayName: Raj Shrivastava
ThreadText: # How do I calculate the gradient of the hinge loss function?
  
  (Asked by: username_0 on 2018-10-06)
  
  With reference to the research paper entitled [Sentiment Embeddings with Applications to Sentiment Analysis](https://ieeexplore.ieee.org/document/7296633), I am trying to implement its sentiment ranking model in Python, for which I am required to optimize the following hinge loss function:
  
  $$\operatorname{loss}\_{\text {sRank}}=\sum\_{t}^{T} \max \left(0,1-\delta\_{s}(t) f\_{0}^{\text {rank}}(t)+\delta\_{s}(t) f\_{1}^{\text {rank}}(t)\right)$$
  
  Unlike the usual mean square error, I cannot find its gradient to perform backpropagation.
  
  How do I calculate the gradient of this loss function?
  
  **Tags:** neural-networks, objective-functions, backpropagation, math, gradient-descent
  
  > **Comments:**
  > **username_1**: You will either need to contact the authors and hope at least one responds with the equation or you need to calculate it yourself.  The following article might help you: https://twice22.github.io/hingeloss/  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-12-18. Score: 1
  
  Hinge loss is difficult to work with when the derivative is needed because the derivative will be a piece-wise function. `max` has one non-differentiable point in its solution, and thus the derivative has the same. This was a very prominent issue with non-separable cases of SVM (and a good reason to use ridge regression).
  
  Here's a slide (Original source from [Zhuowen Tu](https://sites.google.com/site/ucsdfall2018cogs118a/calendar-notes), apologies for the title typo):
  [![SVM Hinge Loss equation and derivative](https://i.sstatic.net/hoaGW.png)](https://i.sstatic.net/hoaGW.png)
  
  Where hinge loss is defined as `max(0, 1-v)` and `v` is the decision boundary of the SVM classifier. More can be found on [the Hinge Loss Wikipedia](https://en.wikipedia.org/wiki/Hinge_loss).
  
  As for your equation: you can easily pick out the `v` of the equation, however without more context of those functions it's hard to say how to derive. Unfortunately I don't have access to the paper and cannot guide you any further...
  
  
  ---
  
  ### Answer by username_3 on 2022-10-02. Score: 1
  
  I disagree with the earlier answer that this is difficult to calculate. If we have the function
  \begin{align\*}
  \sum\_{t\in\mathcal{T}} \max \{0, 1 - d(t) \, y(t, \theta)\}
  \end{align\*}
  the gradient with respect to $\theta$ is
  \begin{align\*}
  & \sum\_{t\in\mathcal{T}}g(t) \\
  & g(t) := \begin{cases}
  0 & \text{ if }1 - d(t) y(t, \theta) < 0 \\
  -d(t)\dfrac{\partial y}{\partial \theta} & \text{ otherwise} \\
  \end{cases}
  \end{align\*}
  Theoretically this is ok, it just means that the gradient is not continuous. However, the objective is still continuous assuming that $d$ and $y$ are both continuous.
  
  In practice, it's not a problem either. Any automatic differentiation software (tensorflow, pytorch, jax) will handle something like this automatically and correctly.
  
  
  ---
  
  