Body: <p>The objective function of an SVM is the following:</p>
  <p><span class="math-container">$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \cdot \mathbf{w}$$</span>
  where</p>
  <ul>
  <li><span class="math-container">$\mathbf{w}$</span> is the model's feature weights and <span class="math-container">$b$</span> is its bias parameter</li>
  <li><span class="math-container">$\mathbf{x}^{(i)}$</span> is the <span class="math-container">$i^\text{th}$</span> training instance's feature vector</li>
  <li><span class="math-container">$y^{(i)}$</span> is the target class (<span class="math-container">$-1$</span> or <span class="math-container">$1$</span>) for the <span class="math-container">$i^\text{th}$</span> instance</li>
  <li><span class="math-container">$m$</span> is the number of training instances</li>
  <li><span class="math-container">$C$</span> is the regularisation hyper-parameter</li>
  </ul>
  <p>And if I was to use a kernel, this would become:</p>
  <p><span class="math-container">$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{u}^{t} \cdot \mathbf{K}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{u}^{t} \cdot \mathbf{K} \cdot \mathbf{u}$$</span></p>
  <p>where the kernel can be the Gaussian kernel:</p>
  <p><span class="math-container">$$K(\mathbf{u}, \mathbf{v})=e^{-\gamma\|\mathbf{u}-\mathbf{v}\|^{2}}$$</span></p>
  <p>How would I go about finding its gradient with respect to the input?</p>
  <p>I need to know this as to then apply this to a larger problem of a CNN with its last layer being this SVM, so I can then find the gradient of this output wrt the input of the CNN.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-11T09:23:12.547Z
Comments.Text: if by `max` you mean the actual maximum between two values, you may be in trouble since the gradient of the maximum is not [smooth](https://en.wikipedia.org/wiki/Smooth_maximum) and hence not differentiable. So you can only compute the gradient as long as you have `1-yy_pred !=0`.
Comments.UserDisplayName: FirefoxMetzger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-11T09:34:09.563Z
Comments.Text: I found some work done on this here: http://www.ecmlpkdd2013.org/wp-content/uploads/2013/07/527.pdf
  They talk about something similar in Section 3.1. Does it hold any weight with what I want to do?
Comments.UserDisplayName: FeedMeInformation
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-16T16:48:55.790Z
Comments.Text: @FeedMeInformation Have you found a solution to this problem?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-18T14:32:48.020Z
Comments.Text: See also https://ai.stackexchange.com/q/8281/2444.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-25T12:07:47.387Z
Comments.Text: Have you found the solution/answer to this problem/question? If yes, feel free to write it below. If not, I will consider starting a bounty on this post, if you still need an answer.
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks><gradient-descent><support-vector-machine>
Title: What is the gradient of a non-linear SVM with respect to the input?
UserDisplayName: FeedMeInformation
ThreadText: # What is the gradient of a non-linear SVM with respect to the input?
  
  (Asked by: username_0 on 2020-04-11)
  
  The objective function of an SVM is the following:
  
  $$J(\mathbf{w}, b)=C \sum\_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \cdot \mathbf{w}$$
  where
  
  * $\mathbf{w}$ is the model's feature weights and $b$ is its bias parameter
  * $\mathbf{x}^{(i)}$ is the $i^\text{th}$ training instance's feature vector
  * $y^{(i)}$ is the target class ($-1$ or $1$) for the $i^\text{th}$ instance
  * $m$ is the number of training instances
  * $C$ is the regularisation hyper-parameter
  
  And if I was to use a kernel, this would become:
  
  $$J(\mathbf{w}, b)=C \sum\_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{u}^{t} \cdot \mathbf{K}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{u}^{t} \cdot \mathbf{K} \cdot \mathbf{u}$$
  
  where the kernel can be the Gaussian kernel:
  
  $$K(\mathbf{u}, \mathbf{v})=e^{-\gamma\|\mathbf{u}-\mathbf{v}\|^{2}}$$
  
  How would I go about finding its gradient with respect to the input?
  
  I need to know this as to then apply this to a larger problem of a CNN with its last layer being this SVM, so I can then find the gradient of this output wrt the input of the CNN.
  
  **Tags:** convolutional-neural-networks, gradient-descent, support-vector-machine
  
  > **Comments:**
  > **username_1**: if by `max` you mean the actual maximum between two values, you may be in trouble since the gradient of the maximum is not [smooth](https://en.wikipedia.org/wiki/Smooth_maximum) and hence not differentiable. So you can only compute the gradient as long as you have `1-yy_pred !=0`.  
  > **username_0**: I found some work done on this here: http://www.ecmlpkdd2013.org/wp-content/uploads/2013/07/527.pdf They talk about something similar in Section 3.1. Does it hold any weight with what I want to do?  
  > **username_2**: @username_0 Have you found a solution to this problem?  
  > **username_2**: See also https://ai.stackexchange.com/q/8281/2444.  
  > **username_2**: Have you found the solution/answer to this problem/question? If yes, feel free to write it below. If not, I will consider starting a bounty on this post, if you still need an answer.  
  
  ---
  
  ## 0 answers
  
  